Between Rigor and Reality: How AI Safety Benchmark Developers Understand Benchmark Use and Maintenance

Megan Li, Ningjing Tang, Christopher Stewart, Hoda Heidari, Hong ShenarXiv 2026 · arXiv

TL;DR

While safety benchmarks are the primary tools used to evaluate generative AI risks, they often fail to meet real-world needs. This paper interviews the developers behind these benchmarks to reveal a stark "disconnect": developers know their tools have a short shelf life and struggle with subjective safety definitions, yet they lack the feedback loops and incentives to maintain them, leading industry practitioners to favor "messy" internal evaluation ecosystems over rigorous public standards.

What Problem Does This Paper Address?

In the rapidly evolving world of Generative AI (GenAI), safety benchmarks—standardized sets of prompts and metrics designed to catch toxicity, bias, or harmfulness—are supposed to be our guardrails. However, recent evidence suggests these benchmarks are often "brittle." They suffer from data leakage (where the test questions end up in the model's training data), concept shift (where societal definitions of "harm" evolve), and a general failure to predict how a model will behave once deployed.

While previous research has focused on the people who use these benchmarks, this paper turns the lens on the developers who build them. The authors investigate why benchmarks exhibit these limitations and why there is such a gap between academic rigor and the "bricolage" (hand-crafted, ad-hoc) reality of industry safety testing.

Methods

The researchers conducted a qualitative sociotechnical study consisting of five semi-structured interviews with AI safety benchmark developers from both academia and industry. These participants had between 1 and 10+ years of experience in areas like mental health, bias, toxicity, and pluralism.

The team used reflexive thematic analysis to process the interview transcripts. This involved an iterative coding process to identify emerging patterns regarding how developers perceive the intended vs. actual use of their tools, how they handle the subjectivity of safety, and how they navigate the "expiration dates" of their datasets.

Datasets

This paper does not introduce a new technical dataset. Instead, it analyzes the practice of dataset creation. The participants discussed their experiences developing benchmarks for:

  • Mental Health: Evaluating model responses to sensitive psychological prompts.
  • Bias and Toxicity: Measuring harmful stereotypes and language.
  • Human Well-being and Pluralism: Capturing diverse perspectives on what constitutes "safety" across different cultures.

Key Insights and Ideas

  • The "Bug vs. Feature" Debate of Subjectivity: Achieving high inter-rater reliability (IRR)—where different human annotators agree on a label—is notoriously hard in safety. Some developers see low agreement as a "bug" to be fixed with more training. Others see it as a "feature," arguing that safety is inherently pluralistic and benchmarks should capture a distribution of human opinions rather than a single "correct" answer.
  • Benchmarks as PR: Some participants expressed skepticism, noting that corporations might use safety benchmarks as a "PR campaign" to give a false sense of security ("our product is okay") without facing clear consequences for poor scores.
  • The Feedback Void: Developers often "throw their benchmarks over the wall" by posting them on platforms like Hugging Face. Because they rarely receive feedback on how (or if) these tools are used in production, they have zero incentive to perform the expensive, time-consuming work of updating them.
  • Bricolage vs. Top-Down Rigor: There is a move away from the "top-down" academic ideal of a perfect, static benchmark. Instead, industry relies on a "bricolage" approach—an evolving ecosystem of red-teaming, user logs, and constant iterative updates that are rarely shared publicly.

Key Results

  • Non-Use by Creators: Ironically, participants in industry contexts reported that they do not use published benchmarks for their own consequential decision-making, preferring internal, adaptive evaluation systems.
  • Maintenance Failure: Despite being aware of threats like data leakage and concept shift, only one out of five participants reported regularly updating their benchmark. Most had no plans to do so due to a lack of resources and adoption signals.
  • The "Expiration Date": One developer estimated that a benchmark prompt is only truly valid until it is shared; once publicized, it is almost immediately captured by web-crawling training sets, rendering it useless for evaluation.

Limitations and Future Work

The primary limitation is the small sample size (n=5), which the authors acknowledge was due to the difficulty of recruiting busy industry practitioners. The sample may also reflect a selection bias, as participants were drawn from the authors' professional networks.

Future Work suggested by the authors includes:

  1. Closing the Feedback Loop: Creating institutional channels (like NIST) to funnel real-world performance data back to benchmark developers.
  2. Methodological Transparency: Encouraging industry to publish high-level descriptions of their internal "bricolage" safety practices.
  3. Collaborative Standards: Developing shared academic-industry norms for when a benchmark should be officially "retired" or "deprecated," similar to how software libraries are managed.

Summary updated

Abstract

Safety benchmarks are cited as the primary tool to understand the risks posed by generative AI (GenAI) systems, yet growing evidence suggests they often fail to meet the needs of real-world safety evaluation. We present findings from five interviews with AI safety benchmark developers in academia and industry. We find a disconnect between perceptions of the intended usefulness of safety benchmarks and their practical usefulness in safety evaluations of deployed systems. Our participants offered perspectives on two challenges contributing to this disconnect: (1) the difficulty of achieving inter-rater reliability on safety constructs, and (2) a lack of clarity regarding how to address persistent threats to external validity. Based on our analysis, we argue that safety benchmarks must not only be grounded in deployment contexts, but actively integrated with them. To facilitate this, we outline pathways for more transparent communication between academic and industry AI safety stakeholders.
BibTeX
@misc{li2026between,
  author = {Megan Li and Ningjing Tang and Christopher Stewart and Hoda Heidari and Hong Shen},
  title = {{Between Rigor and Reality: How AI Safety Benchmark Developers Understand Benchmark Use and Maintenance}},
  year = {2026},
  eprint = {2611.18930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18930},
}
Figure