In Benchmarks We Trust ... Or Not?

Ine Gevers, Victor De Marez, Jens Van Nooten, Jens Lemmens, Andriy Kosar, Ehsan Lotfi, Nikolay Banar, Pieter Fivez, Luna De Bruyne, Walter DaelemansEMNLP 2025

TL;DR

This paper critically examines the current state of Natural Language Processing (NLP) benchmarks, arguing that while essential for progress, they are increasingly failing to accurately assess the capabilities of modern Large Language Models (LLMs). The authors propose concrete checklists for both creating and using benchmarks to improve their integrity and validity, ultimately aiming for more reliable evaluations and generalizable model performance.

What Problem Does This Paper Address?

The NLP field heavily relies on standardized benchmarks to evaluate and compare models. However, the rapid advancement of LLMs has exposed significant shortcomings in these benchmarks. Models are "outpacing" benchmark creation, leading to saturated benchmarks that don't necessarily reflect true understanding. This has created a situation where benchmark scores can become a goal in themselves, potentially compromising research integrity through "gaming" or "shortcut learning." Furthermore, existing benchmarks often suffer from issues like data contamination, Anglocentric bias, and a lack of robustness, leading to unreliable and misleading evaluations of model capabilities. The paper aims to address this crisis of confidence in NLP benchmarks by identifying the core problems and proposing actionable solutions.

Methods

The authors conducted a comprehensive survey of existing literature on NLP benchmarking issues, categorizing them into four types of experimental validity: internal, external, statistical, and construct validity. They then surveyed proposed solutions in the literature for each identified issue. Based on this analysis, they developed two concrete checklists: one for constructing benchmarks and another for evaluating methods on existing benchmarks. These checklists are designed to cover a wide range of issues, from data quality and bias to evaluation methodology and transparency. The paper illustrates the application of these checklists by evaluating three popular NLP benchmarks: SuperGLUE, WinoGrande, and ARC-AGI.

Datasets

The paper does not introduce new datasets. Instead, it focuses on analyzing and proposing improvements for existing NLP benchmarks. The authors specifically mention SuperGLUE, WinoGrande, and ARC-AGI as examples used to illustrate their proposed checklists. They also reference various other benchmarks and datasets throughout their discussion of specific issues, such as HellaSwag, TriviaQA, LegalBench, and FinBen.

Key Insights and Ideas

  • Benchmarks are becoming "gameable": LLMs can exploit superficial patterns or "dataset artifacts" to achieve high scores without genuine understanding, leading to inflated performance metrics.
  • Data contamination is a pervasive threat: Models are often exposed to benchmark data during training, leading to artificially high scores and unfair comparisons. This leakage can occur in various forms and across different training stages.
  • Existing benchmarks are often Anglocentric and lack diversity: The heavy focus on English and high-resource languages overlooks the vast linguistic landscape, leading to biased evaluations for LLMs.
  • Construct validity is a major concern: Benchmarks often serve as flawed proxies for general abilities, creating an "illusion of generality" where models perform well on the benchmark but fail in real-world scenarios.
  • A "one-size-fits-all" solution is unlikely: The paper emphasizes that addressing benchmarking issues requires a holistic approach, combining multiple strategies rather than relying on single fixes.
  • Downstream test-suites are crucial: The authors propose integrating minimal-sized test-suites for real-life use cases to complement traditional benchmarks, ensuring better generalizability of model performance.

Key Results

The paper's primary contribution is not a set of quantitative results from a new model or dataset, but rather a structured analysis of benchmarking flaws and a proposed framework for improvement. The evaluation of SuperGLUE, WinoGrande, and ARC-AGI using their checklist (presented in Table 3) reveals that even these popular benchmarks fall short on several key criteria, particularly regarding detailed annotation metadata, authentic data in diverse languages, and avoiding memorized data. This highlights the widespread nature of the identified problems.

Limitations and Future Work

The authors acknowledge several limitations:

  • Incompleteness: Compiling an exhaustive list of all benchmarking issues and solutions is challenging, and some might have been missed.
  • Interdependencies of solutions: The paper notes a lack of research on how different proposed solutions might interact or potentially exacerbate other issues.
  • Generalizability of checklists: The proposed checklists are presented as guidelines and may need adaptation for specific benchmarking scenarios.
  • Text modality focus: The study primarily focuses on text-based benchmarks, acknowledging that similar issues exist in other modalities like vision.

For future work, the authors strongly suggest creating minimal-sized test-suites for real-life use cases to complement NLP benchmarking. They also advocate for further research into the interdependencies of various benchmarking solutions and the development of more robust and adaptable evaluation frameworks.

Summary updated

BibTeX
@inproceedings{gevers2025benchmarks,
  author = {Ine Gevers and Victor De Marez and Jens Van Nooten and Jens Lemmens and Andriy Kosar and Ehsan Lotfi and Nikolay Banar and Pieter Fivez and Luna De Bruyne and Walter Daelemans},
  title = {{In Benchmarks We Trust ... Or Not?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure