Based on the observation that sports teams rely on colored jerseys to define group membership, we examined how grouping by similarity affected observers’ abilities to track a “ball” target passed between 20 colored circle “players” divided into two color “teams” of 10 players each, or five color teams of four players each. Observers were more accurate and exerted less effort (indexed by pupil diameter) when their task was to count the number of times any player gained possession of the ball versus when they had to count only the possessions by a given color team, especially when counting the possessions of one team when players were grouped into fewer teams of more individual members each. Overall, results confirm previous reports of costs for segregating a larger set into smaller subsets and suggest that grouping by similarity facilitates processing at the set level.