What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks

Meera Desai, Sang T. Truong, Hanna Wallach, Alex Chouldechova, A. Feder Cooper, Jean Garcia-Gathright, Daniel E. Ho, Abigail Z. Jacobs, Sanmi Koyejo, Nicholas Pangakis, Angelina Wang

TL;DR

AI benchmarks are heavily relied upon to evaluate model capabilities and safety, but do they actually measure what they claim to? This paper adapts validation frameworks from the social sciences to evaluate 56 popular AI benchmarks across 53 models. The findings reveal a systemic issue: safety benchmarks often disagree with each other, capability benchmarks fail to differentiate from one another, and benchmark design choices (like formatting) frequently overshadow the actual concept being tested.

What Problem Does This Paper Address?

When a new LLM drops, its release is usually accompanied by a scorecard showing high marks on benchmarks for "reasoning," "knowledge," or "bias." But how do we know a reasoning benchmark is actually measuring reasoning, and not just memorization or compliance with a specific prompt format?

This is the problem of construct validity—ensuring an evaluation tool truly measures the abstract concept it purports to. Currently, AI benchmarking lacks rigorous validation standards. If two benchmarks both claim to measure "bias" but rank models completely differently, or if a "reasoning" benchmark ranks models identically to a "knowledge" benchmark, our evaluations are deeply flawed. This paper introduces a systematic approach to interrogate whether AI benchmarks are actually measuring distinct, coherent concepts.

Methods

The authors borrow and adapt two core concepts from psychometrics and the social sciences:

  • Convergence: Do different benchmarks that claim to measure the same concept (e.g., reasoning) rank models similarly?
  • Discrimination: Do benchmarks measuring different concepts (e.g., reasoning vs. safety) produce distinct model rankings?

They mapped 56 benchmarks into high-level "assigned concepts" (4 capability concepts, 7 safety concepts) and analyzed them at two levels:

  1. Benchmark-level: They computed Spearman rank correlations between model scores across all pairs of benchmarks to see how well the rankings aligned.
  2. Item-level: They applied Item Response Theory (IRT) (specifically a 1-parameter logistic model) to test whether a model's item-by-item responses across two benchmarks are better predicted by a single shared latent trait or by separate, benchmark-specific traits.

Additionally, they used hierarchical clustering and partial Mantel tests to detect method effects—whether shared design elements (such as multiple-choice vs. free-response scored by an LLM-judge) cause benchmarks to cluster together regardless of the underlying concept.

Datasets

The authors compiled an extensive dataset consisting of outputs and scores from 53 instruction-tuned models evaluated across 56 capability and safety benchmarks (sourced from initial suites like HELM and SafetyPrompts).

After filtering out benchmarks that were "saturated" (where top models clustered too closely to differentiate performance) or suffered from formatting compliance issues, they focused on 48 benchmarks for benchmark-level analysis and 37 benchmarks for item-level IRT analysis. The complete dataset of model outputs and item-level scores has been released on HuggingFace.

Key Insights and Ideas

  • Safety benchmarks are highly inconsistent: Benchmarks for safety concepts like refusal, bias, and safety detection show very weak convergence. This suggests these safety concepts are either inherently multi-dimensional or defined inconsistently across datasets.
  • Capability benchmarks fail to discriminate: Model rankings on reasoning benchmarks are almost perfectly correlated with rankings on knowledge benchmarks. Rather than measuring distinct traits, these benchmarks largely track a single, dominant underlying capability factor.
  • Format over substance: How you ask a question can matter more than what you ask. Benchmarks that use an LLM-judge cluster together tightly in model rankings, even if one is measuring refusal and another is measuring an entirely different safety concept.
  • Mislabeled benchmarks measure the wrong thing: Some prominent benchmarks track entirely different concepts than advertised. For instance, BBQ-accuracy (a widely used bias benchmark) actually tracks model reasoning capabilities much better than it tracks bias.

Key Results

  • Convergence vs. Discrimination: As shown in the correlation matrix, capability benchmarks (green box, top-left) correlate strongly both within and across concepts, meaning they fail to discriminate well. Conversely, safety benchmarks (orange box, bottom-right) show much weaker correlations overall.
Figure 2: Benchmarks labeled with capability concepts (green, upper left) overall exhibit stronger relationships across assigned concepts than benchmarks labeled with safety con- cepts (orange, bottom right). Cells indicate pairwise average Spearman correlations between model rankings on benchmarks by assigned concepts, with gray 95% cluster bootstrap con- fidence intervals (resampled by model family). Marked concepts (*) are inverted so that positive scores indicate desirable behavior throughout.
  • Capability vs. Safety Distributions: Looking at the pairwise correlations directly, capability benchmarks consistently maintain high correlation coefficients, while safety benchmarks like refusal, safety detection, and bias exhibit wide variances that frequently drop near or below zero.
Figure 1: Convergence of benchmarks with the same assigned concept (n = number of benchmarks per assigned concept). Distributions of pairwise Spearman correlations be- tween model rankings on benchmarks with the same assigned concept; model rankings on benchmarks labeled with capability concepts correlate consistently with one another, while model rankings on benchmarks labeled with safety concepts do not.
  • Quantifying Method Effects: A partial Mantel test revealed that when isolating free-response benchmarks scored by an LLM-judge, the choice of score format was a dominant predictor of benchmark similarity ($\beta_{format} = 0.526, p < 0.0001$), completely washing out the effect of the assigned concept ($\beta_{concept} = -0.058, p = 0.998$).
  • The BBQ Case Study: A permutation relabeling test statistically confirmed that BBQ-accuracy correlates more strongly with reasoning benchmarks than with other bias benchmarks (relabeling statistic = 0.15, $p < 0.001$). This is critical because BBQ is often the only bias metric reported in major commercial model releases.

Limitations and Future Work

  • Underspecified Concepts: Because original benchmark developers rarely provide precise, formal definitions of concepts like "reasoning" or "bias," it remains difficult to determine if a low correlation signifies a broken benchmark or merely a different, valid interpretation of the concept.
  • The "General Intelligence" Factor: The study's discriminant analysis does not explicitly control for the single dominant dimension of general model capability that drives performance across almost all tasks. The authors suggest that future work should account for this by residualizing benchmarks against the first principal component of the correlation matrix.
  • IRT Assumptions: The item-level IRT models assume benchmarks are unidimensional, an assumption that likely violates the reality of complex, multi-dimensional safety benchmarks.

Summary updated

Abstract

Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal). We adapt convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, applying it to 56 capability and safety benchmarks across 53 models. We label benchmarks with substantively similar purported concepts to a shared assigned concept, and ask whether model rankings on benchmarks with the same assigned concept correlate more strongly than rankings on benchmarks with different assigned concepts. We ask analogous questions at the item level using item response theory (IRT) models. We find that correlations between model rankings on benchmarks with the same assigned safety concepts are often weak, suggesting these concepts may be conceptualized inconsistently across benchmarks. For assigned capability concepts (e.g., reasoning, knowledge), model rankings are often as strongly correlated among benchmarks with the same assigned concept as between benchmarks with different assigned concepts, suggesting these capability concepts may not discriminate well from one another. In some cases, benchmarks that share design elements (e.g., score format) correlate more strongly than benchmarks with the same assigned concept. Finally, some individual benchmarks correlate more strongly with benchmarks assigned a different concept than with benchmarks sharing their own assigned concept, suggesting they may measure a different concept than they purport to. For example, BBQ-accuracy correlates more strongly with benchmarks labeled reasoning than with benchmarks that share its assigned concept, bias. To support future empirical work on benchmark validity, we release our extensive dataset of model outputs and scores at the item- and benchmark-level.
BibTeX
@inproceedings{desai2026what,
  author = {Meera Desai and Sang T. Truong and Hanna Wallach and Alex Chouldechova and A. Feder Cooper and Jean Garcia-Gathright and Daniel E. Ho and Abigail Z. Jacobs and Sanmi Koyejo and Nicholas Pangakis and Angelina Wang},
  title = {{What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=889XnQKyhM},
}
Figure