TL;DR
AI benchmarks are heavily relied upon to evaluate model capabilities and safety, but do they actually measure what they claim to? This paper adapts validation frameworks from the social sciences to evaluate 56 popular AI benchmarks across 53 models. The findings reveal a systemic issue: safety benchmarks often disagree with each other, capability benchmarks fail to differentiate from one another, and benchmark design choices (like formatting) frequently overshadow the actual concept being tested.
What Problem Does This Paper Address?
When a new LLM drops, its release is usually accompanied by a scorecard showing high marks on benchmarks for "reasoning," "knowledge," or "bias." But how do we know a reasoning benchmark is actually measuring reasoning, and not just memorization or compliance with a specific prompt format?
This is the problem of construct validity—ensuring an evaluation tool truly measures the abstract concept it purports to. Currently, AI benchmarking lacks rigorous validation standards. If two benchmarks both claim to measure "bias" but rank models completely differently, or if a "reasoning" benchmark ranks models identically to a "knowledge" benchmark, our evaluations are deeply flawed. This paper introduces a systematic approach to interrogate whether AI benchmarks are actually measuring distinct, coherent concepts.
Methods
The authors borrow and adapt two core concepts from psychometrics and the social sciences:
- Convergence: Do different benchmarks that claim to measure the same concept (e.g., reasoning) rank models similarly?
- Discrimination: Do benchmarks measuring different concepts (e.g., reasoning vs. safety) produce distinct model rankings?
They mapped 56 benchmarks into high-level "assigned concepts" (4 capability concepts, 7 safety concepts) and analyzed them at two levels:
- Benchmark-level: They computed Spearman rank correlations between model scores across all pairs of benchmarks to see how well the rankings aligned.
- Item-level: They applied Item Response Theory (IRT) (specifically a 1-parameter logistic model) to test whether a model's item-by-item responses across two benchmarks are better predicted by a single shared latent trait or by separate, benchmark-specific traits.
Additionally, they used hierarchical clustering and partial Mantel tests to detect method effects—whether shared design elements (such as multiple-choice vs. free-response scored by an LLM-judge) cause benchmarks to cluster together regardless of the underlying concept.
Datasets
The authors compiled an extensive dataset consisting of outputs and scores from 53 instruction-tuned models evaluated across 56 capability and safety benchmarks (sourced from initial suites like HELM and SafetyPrompts).
After filtering out benchmarks that were "saturated" (where top models clustered too closely to differentiate performance) or suffered from formatting compliance issues, they focused on 48 benchmarks for benchmark-level analysis and 37 benchmarks for item-level IRT analysis. The complete dataset of model outputs and item-level scores has been released on HuggingFace.
Key Insights and Ideas
- Safety benchmarks are highly inconsistent: Benchmarks for safety concepts like refusal, bias, and safety detection show very weak convergence. This suggests these safety concepts are either inherently multi-dimensional or defined inconsistently across datasets.
- Capability benchmarks fail to discriminate: Model rankings on reasoning benchmarks are almost perfectly correlated with rankings on knowledge benchmarks. Rather than measuring distinct traits, these benchmarks largely track a single, dominant underlying capability factor.
- Format over substance: How you ask a question can matter more than what you ask. Benchmarks that use an LLM-judge cluster together tightly in model rankings, even if one is measuring refusal and another is measuring an entirely different safety concept.
- Mislabeled benchmarks measure the wrong thing: Some prominent benchmarks track entirely different concepts than advertised. For instance,
BBQ-accuracy(a widely used bias benchmark) actually tracks model reasoning capabilities much better than it tracks bias.
Key Results
- Convergence vs. Discrimination: As shown in the correlation matrix, capability benchmarks (green box, top-left) correlate strongly both within and across concepts, meaning they fail to discriminate well. Conversely, safety benchmarks (orange box, bottom-right) show much weaker correlations overall.
- Capability vs. Safety Distributions: Looking at the pairwise correlations directly, capability benchmarks consistently maintain high correlation coefficients, while safety benchmarks like refusal, safety detection, and bias exhibit wide variances that frequently drop near or below zero.
- Quantifying Method Effects: A partial Mantel test revealed that when isolating free-response benchmarks scored by an LLM-judge, the choice of score format was a dominant predictor of benchmark similarity ($\beta_{format} = 0.526, p < 0.0001$), completely washing out the effect of the assigned concept ($\beta_{concept} = -0.058, p = 0.998$).
- The BBQ Case Study: A permutation relabeling test statistically confirmed that
BBQ-accuracycorrelates more strongly with reasoning benchmarks than with other bias benchmarks (relabeling statistic = 0.15, $p < 0.001$). This is critical because BBQ is often the only bias metric reported in major commercial model releases.
Limitations and Future Work
- Underspecified Concepts: Because original benchmark developers rarely provide precise, formal definitions of concepts like "reasoning" or "bias," it remains difficult to determine if a low correlation signifies a broken benchmark or merely a different, valid interpretation of the concept.
- The "General Intelligence" Factor: The study's discriminant analysis does not explicitly control for the single dominant dimension of general model capability that drives performance across almost all tasks. The authors suggest that future work should account for this by residualizing benchmarks against the first principal component of the correlation matrix.
- IRT Assumptions: The item-level IRT models assume benchmarks are unidimensional, an assumption that likely violates the reality of complex, multi-dimensional safety benchmarks.
Summary updated