TL;DR
This paper investigates how to best select individuals for a team to maximize collective performance. The authors demonstrate that traditional "canonical" tests measuring individual performance are often ineffective for team selection, especially for complex team performance metrics. Instead, they introduce a novel "potential" test that can consistently select near-optimal teams, offering a constant-factor approximation to the best possible team performance.
What Problem Does This Paper Address?
The core problem this paper tackles is team selection: how do we choose the best individuals to form a team that will perform optimally on a given task? This is a fundamental question in social sciences, economics, and organizational behavior. A common approach is to test individuals in isolation and then select those with the highest scores. However, influential work by Hong and Page has shown that this simple strategy can fail, particularly when team performance relies on diverse perspectives. The authors aim to understand why this happens and to develop more effective methods for team selection using individual tests, especially for complex team performance measures. They explore the limits of individual testing and identify specific types of team performance functions where individual tests are inherently insufficient.
Methods
The paper introduces a formal framework for analyzing team performance and individual testing. They define team performance as a set function $g(T)$, where $T$ is a team. They consider various forms of $g(T)$, including:
- Cumulative effects: Team performance is the sum of individual contributions (modular functions).
- Contests: Team performance is the maximum individual contribution (submodular functions).
- Complementarity: Team performance grows with diverse perspectives (submodular functions).
- Synergy: Team performance benefits from interactions (supermodular functions).
The key innovation is the introduction of a "potential" test, denoted $f_h(X)$, which measures an individual's potential rather than their direct performance. This test involves evaluating the expected maximum of multiple independent trials of an individual's performance. The paper then mathematically proves that for "contest" type team performance functions (specifically, the expected maximum or sum of top $h$ performers), this potential test can select a team that achieves a constant factor of the optimal performance, regardless of team size. They also demonstrate that for submodular and supermodular functions, no such individual test can guarantee near-optimal team performance.
Datasets
Not discussed in the paper. The paper uses theoretical models and random variables to represent individuals and their performance, rather than empirical datasets.
Key Insights and Ideas
- The "Canonical Test" is Often Flawed: Simply testing individuals for their direct performance and picking the top scorers is surprisingly ineffective for many team performance models, especially those that benefit from diversity or complex interactions.
- "Potential" is Key for Certain Teams: For team performance metrics based on the maximum individual output (like in contests), testing an individual's potential – their best possible outcome over multiple hypothetical attempts – is a much more effective strategy than testing their average performance.
- Submodularity and Supermodularity Pose Challenges: Team performance functions that are submodular (like those capturing complementarity or diversity) or supermodular (like those capturing synergy) are inherently difficult to optimize using only individual tests. The interdependencies within these functions mean that individual evaluations don't capture the full picture.
- Limits to Individual Testing: There's a fundamental limit to how well individual tests can approximate optimal team performance. For certain types of team performance functions, no individual test can guarantee a constant factor approximation.
Key Results
- Constant Factor Approximation for Contest-like Teams: The paper proves that a specific "potential" test ($f_h(X)$) can select a team that achieves at least a constant fraction (independent of team size) of the optimal team performance for "contest" structures (e.g., expected maximum of individual performances).
- Failure of Individual Tests for Submodular/Supermodular Functions: The authors demonstrate that for certain submodular (e.g., set cardinality) and supermodular (e.g., edge count in a graph) team performance functions, no individual test can guarantee even a constant factor approximation to the optimal team.
- Limits on Test Resolution: For pairwise competitions, a test score can only guarantee a win probability of at least 1/4 against another competitor with a higher score, establishing a fundamental limit on how well scores can predict competitive outcomes.
Limitations and Future Work
- Assumptions on Distributions: The results often rely on knowing the distributions of individual candidate performances. Real-world scenarios might involve noisy samples or unknown distributions, which is a direction for future work.
- Specific Team Performance Models: While the paper covers several classes of team performance functions, exploring other models and their amenability to individual testing remains an open area.
- Characterizing Submodular Functions: A key open question is to precisely characterize which submodular functions allow for good approximations via individual tests.
- Robustness to Noisy Data: Investigating how to select teams when only noisy samples of individual performance are available is a promising avenue.
- Competition Models: Extending the analysis of test scores in competitive settings to more complex scenarios beyond pairwise competitions is also suggested.
Summary updated