ECBD: Evidence-Centered Benchmark Design for NLP

Yu Lu Liu, Su Lin Blodgett, Jackie C. K. Cheung, Vera Liao, Alexandra Olteanu, Ziang XiaoACL 2024 · DOI

Abstract

In the rapidly evolving field of Natural Language Processing (NLP), benchmarks like GLUE, SuperGLUE, and HELM serve as the primary metrics for model success. However, the link between high scores and actual model capabilities—such as "reasoning" or "understanding"—remains frequently unproven. This paper introduces ECBD: Evidence-Centered Benchmark Design, a formal framework adapted from educational testing to bring scientific rigor to NLP evaluation. ECBD moves benchmarking away from simple score reporting toward a systematic process of defining capabilities, justifying design choices, and gathering empirical evidence that a benchmark measures what it claims to measure. Through meta-analyses of existing benchmarks, we identify a systemic "validity gap" and provide a roadmap for more transparent and scientifically valid AI evaluation.

Introduction: The Validity Gap in NLP

Benchmarking in NLP has transitioned from simple, single-task datasets to massive "holistic" evaluations that attempt to measure a model's total intelligence. As these benchmarks grow more ambitious, the relationship between a numerical score and a specific capability becomes increasingly murky. In fields like psychology or education, test design requires formal proof that the instrument measures the intended trait (e.g., ensuring a math test measures calculus skills rather than reading speed). In NLP, datasets and metrics are often selected based on convenience or standard practice rather than formal justification. This lack of rigor creates a "validity gap," where benchmarks are built on a foundation of tacit, untested assumptions.

The ECBD Framework

To address these issues, the authors propose the ECBD framework, inspired by Evidence-Centered Design (ECD) used in high-stakes educational assessments. ECBD decomposes benchmark design into five distinct modules, each requiring a rigorous process of documentation and justification.

The ECBD framework modules and the Describe-Justify-Support workflow.

The Five Modules of ECBD

  1. Capability Module: Explicitly defines the "constructs" (e.g., toxicity, reasoning) the benchmark aims to measure and relates them to the benchmark's intended use.
  2. Content Module: Specifies the pool of test items and justifies how these items elicit the desired capabilities.
  3. Adaptation Module: Describes how the model is prepared for the test (e.g., few-shot prompting vs. fine-tuning) to ensure the test conditions do not unfairly disadvantage certain architectures.
  4. Assembly Module: Explains the logic behind selecting specific items from the pool to create the final test set.
  5. Evidence Module: Split into Extraction (how a response is obtained) and Accumulation (how responses are aggregated into a final score).

For every module, benchmark creators are tasked with a three-step workflow: Describe the design decisions, Justify them with a hypothesis (why these decisions enable the module to fulfill its role), and provide Support (validity evidence that the hypothesis holds true).

Key Insights and Ideas

Through the lens of ECBD, the researchers performed meta-analyses on three major benchmarks: BoolQ, SuperGLUE, and HELM. Their analysis revealed several conceptual pitfalls common in current practices.

Construct vs. Measurement

A recurring issue is the conflation of the thing being measured (the construct) with the method of measurement (the metric). For instance, HELM defines "accuracy" as an "umbrella term for the standard accuracy-like metric," which circularizes the definition and obscures the specific capability being tested.

Hierarchical constructs in HELM and SuperGLUE benchmarks.

Implicit Sub-capabilities and Re-purposing

Benchmarks like SuperGLUE aim for "General Language Understanding" but rely on sub-tasks like "Causal Reasoning" to represent that goal. ECBD highlights that the link between these sub-tasks and the overarching goal is often an untested assumption. Furthermore, the framework warns against the "risk of re-purposing," where old datasets (like BoolQ) are used for new purposes (like measuring social bias) without new validation to ensure the data actually elicits that specific capability.

Results: Systemic Issues in Benchmarking

The analysis of BoolQ, SuperGLUE, and HELM revealed three primary systemic failures:

  • Lack of Justification: While creators are adept at describing what they did, they rarely provide a scientific justification for why those choices are valid for the stated goals.
  • Missing Validity Evidence: Almost none of the analyzed benchmarks provided empirical or theoretical evidence—such as correlation with human expert performance—to support their choice of metrics or data splits.
  • Inconsistent Adaptation: Many benchmarks fail to prescribe specific prompting methods, leading to "noisy" results where a model's failure may stem from poor prompt engineering rather than a lack of underlying capability.

Conclusion and Future Work

While the ECBD process introduces a significant documentation burden, it provides a necessary roadmap for moving NLP evaluation from a culture of "move fast and break things" to one of scientific accountability. By treating NLP models more like students taking an exam and less like engines being tested for raw horsepower, ECBD enables more meaningful and transparent interpretations of leaderboard rankings. Future work involves extending this framework to multi-modal benchmarks and conducting user studies to evaluate how ECBD-informed documentation changes researcher behavior.

Summary updated

BibTeX
@inproceedings{liu2024ecbd,
  author = {Yu Lu Liu and Su Lin Blodgett and Jackie C. K. Cheung and Vera Liao and Alexandra Olteanu and Ziang Xiao},
  title = {{ECBD: Evidence-Centered Benchmark Design for NLP}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.861},
}
Figure