Team Performance with Test Scores

Jon Kleinberg, Maithra Raghu

TL;DR

While standard organizational wisdom suggests that testing individuals in isolation fails to build optimal teams, this paper shows that a clever shift in what we test changes everything. By measuring a candidate's "upside potential" (their best performance over multiple attempts) rather than their average performance, we can mathematically guarantee near-optimal team selection for contest-like environments. However, the authors also prove that when teams require deep synergy or highly specific complementary perspectives, evaluating individuals in isolation is fundamentally doomed to fail.

What Problem Does This Paper Address?

How do you select the perfect team? Imagine you have a large pool of job applicants and need to pick a team of size $k$ to maximize collective performance. The most common real-world approach is to test everyone individually and hire the top scorers.

However, seminal work in the social sciences (notably by Hong and Page) demonstrated that this "canonical test" fails because it ignores team dynamics. Top individual scorers often possess identical skill sets, leading to a homogenous team that lacks the diversity needed to solve complex problems.

This paper tackles a fundamental theoretical question: Is the failure of individual testing an unavoidable fact of team formation, or have we just been using the wrong types of tests? The authors explore whether there are alternative individual scoring metrics that can successfully predict and optimize team performance without needing to test candidates in groups.

Methods

The authors formalize team selection as a set function maximization problem under three distinct structural models:

  1. Contests: Team performance is determined by its top performers (e.g., the expected maximum or the sum of the top $h$ outcomes within the team).
  2. Complementarity: Performance scales based on the number of distinct perspectives or unique skills covered.
  3. Synergy: Performance relies heavily on pairwise interactions between members.

To solve the selection problem for the Contest model, the authors introduce a novel individual testing function called the Potential Test ($f_h$). Instead of measuring a candidate's expected average performance, the potential test calculates the expected maximum outcome if the candidate were allowed to attempt the task multiple times independently.

Mathematically, they use quantile analysis to split a candidate's performance distribution into "top values" and "tail values." By bounding these components separately, they prove that ranking candidates by this "upside potential" yields a team with provable performance guarantees.

Datasets

Not discussed in the paper. This is a purely theoretical paper in mathematical social sciences and theoretical computer science; it relies on rigorous mathematical proofs and synthetic probabilistic counterexamples rather than empirical datasets.

Key Insights and Ideas

  • Average vs. Potential: In a contest setting (like an R&D department or a crowdsourced competition), you don't care about a worker's average day; you care about their absolute best day. Standard tests look at the mean, whereas the optimal test looks at the variance and the right tail of the distribution.
  • The Mathematical Line of Isolation: The paper draws a sharp line between where individual testing works and where it breaks down. If team performance is submodular (contests), individual testing can work if you measure potential. But if the team function relies on complex combinatorial structures—like covering a set of perspectives (Set Cardinality) or spanning a vector space (Linear Matroid Rank)—evaluating people in isolation is mathematically guaranteed to fail.
  • The Non-Transitive Dice Paradox: When moving from teams to pairwise competitions, human performance behaves like non-transitive dice (where Die A beats B, B beats C, and C beats A). Because of this inherent circularity, the authors show that the best possible isolated score can only guarantee a candidate a $\frac{1}{4}$ probability of winning, a bound achieved simply by evaluating a candidate's median performance.

Key Results

  • Constant-Factor Approximation: For contest-style team functions, selecting the top scorers using the Potential Test guarantees a team performance within a constant factor ($\lambda < 30$) of the absolute true optimum, completely independent of the team size $k$. In contrast, the standard average-based test can perform worse by a factor of $k$.
  • Hardness Limit: The authors prove that no individual test can guarantee an approximation ratio closer than $\frac{9}{8} (1.125)$ for the expected maximum.
  • Greedy Optimality: For candidates modeled as weighted Bernoulli random variables (an "on-off" success pattern), a simple greedy hill-climbing algorithm is proven to find the exact optimal team.
  • The Competition Bound: In pairwise matches, mapping a random variable to its median guarantees that if $f(X) \ge f(Y)$, then $P(X \ge Y) \ge \frac{1}{4}$. This $\frac{1}{4}$ bound is mathematically tight.

Limitations and Future Work

  • Perfect Distribution Knowledge: The proposed potential test assumes that the evaluator knows the exact probability distribution of each candidate's performance. In reality, employers only have access to noisy, limited samples (e.g., a few interview questions or past portfolio pieces).
  • Independence Assumption: The positive results for contest environments rely heavily on the assumption that candidates' performances are mutually independent. If candidates' performances are highly correlated, the guarantees may break down.
  • Open Question: The authors suggest future work should focus on characterizing the exact boundary of submodular functions that allow for individual test approximations, potentially linking this domain to the literature on "juntas" and robust estimation.

Summary updated

Abstract

Team performance is a ubiquitous area of inquiry in the social sciences, and it motivates the problem of team selection -- choosing the members of a team for maximum performance. Influential work of Hong and Page has argued that testing individuals in isolation and then assembling the highest-scoring ones into a team is not an effective method for team selection. For a broad class of performance measures, based on the expected maximum of random variables representing individual candidates, we show that tests directly measuring individual performance are indeed ineffective, but that a more subtle family of tests used in isolation can provide a constant-factor approximation for team performance. These new tests measure the "potential" of individuals, in a precise sense, rather than performance, to our knowledge they represent the first time that individual tests have been shown to produce near-optimal teams for a non-trivial team performance measure. We also show families of subdmodular and supermodular team performance functions for which no test applied to individuals can produce near-optimal teams, and discuss implications for submodular maximization via hill-climbing.
BibTeX
@article{kleinberg2015team,
  author = {Jon Kleinberg and Maithra Raghu},
  title = {{Team Performance with Test Scores}},
  journal = {ACM Transactions on Economics and Computation},
  year = {2015},
}
Figure