Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

Christopher M. Stewart, Preston Botter, Natalie Sarabosing, Muye Zhang, Rachel Phinnemore, Shalini Ghosh, Hong Shen, Hoda HeidariarXiv 2026 · arXiv

TL;DR

This paper applies rigorous psychometric auditing to AI safety benchmarks, revealing that "harmful refusal" is not a single, unified trait. By analyzing the HarmBench dataset, the authors demonstrate that aggregating diverse safety behaviors into a single top-line score masks critical differences in model performance and developer-specific failure patterns.

What Problem Does This Paper Address?

AI safety leaderboards often boil down complex model behaviors into a single "safety score." This practice relies on an unstated assumption of construct validity: that there is a single, measurable attribute (like "harmful refusal") driving a model's responses across all test items.

If this assumption is false—if "safety" is actually a collection of unrelated behaviors—then a single score is misleading. Two models could achieve the same score while having completely different safety profiles (e.g., one is great at refusing cybercrime prompts but terrible at copyright, while the other is the opposite). The authors set out to audit whether the "harmful refusal" construct in the popular HELM Safety benchmark actually holds up under statistical scrutiny.

Methods

The authors treat AI benchmarks as psychological tests and apply two key techniques from the field of psychometrics:

  1. Multidimensional Item Response Theory (MIRT): Standard benchmarks assume a 1D "ability" axis. MIRT allows for multiple axes (dimensions). The authors compared a simple 1D model against multidimensional models (e.g., a 3D model separating "standard," "contextual," and "copyright" prompts) to see which better predicted how models actually pass or fail items.
  2. Differential Item Functioning (DIF): This technique looks for "construct-irrelevant variance." If "harmful refusal" is the only thing being measured, then two models with the same overall score should have the same probability of passing any specific item, regardless of who built them. If OpenAI models consistently fail an item that Anthropic models of equal "ability" pass, that item is flagged for DIF, suggesting it's measuring something other than the intended construct.
Figure 1: The HarmBench score supports a claim about a model only through a warrant. The warrant holds that one harmful refusal construct organizes the item responses. Our two construct validity tests probe that warrant, one from inside the response matrix and one from outside it.

Datasets

The study analyzed the HELM Safety (v1.17.0) model pool, which includes 81 models. They initially looked at four datasets targeting harmful refusal:

  • HarmBench: (400 items) The primary focus of the study.
  • SimpleSafetyTests, AnthropicRedTeam, and XSTest (harmful subset): These were found to be "saturated," meaning modern models pass them so consistently (92-94% pass rates) that they provide little signal for item-level psychometric analysis.

Key Insights and Ideas

  • Safety is not one-dimensional: The data strongly rejects the idea that a single "refusal" trait explains model behavior. Models that are good at refusing direct harmful requests aren't necessarily the same ones good at refusing to generate copyrighted text.
  • Copyright is an outlier: Copyright items separate cleanly from other harm domains. The authors argue this is because copyright refusal is driven more by memorization during training than by a general "refusal disposition." Lumping them together is like averaging a student's math and history scores into a single "intelligence" metric.
  • The "Developer Signature": When using a single aggregate score, models from different developers (OpenAI vs. Anthropic) show significant differences in how they handle specific items. These differences mostly disappear when you match models on more specific sub-scores (like "cybercrime" or "harassment"), suggesting that developer-specific training "signatures" are being flattened and hidden by top-line averages.

Key Results

  • Predictive Power: Multidimensional models significantly outperformed the standard 1D model. As shown in Table 1, moving from a 1D model to a 3D model (separating response processes) dropped the held-out log-loss from 0.470 to 0.258, a massive improvement in explaining model behavior.
Table 1: HarmBench model comparison across five repeated 80/20 response-level holdout splits. Log-loss and Brier entries are mean (SD) across splits after selecting one fit per model and split from twenty random starts by training log-likelihood; held-out responses were not used for restart selection. All models are 2PL except the unidimensional 3PL row included as the strongest unidimensional comparator. Exploratory models allow all items to load on all dimensions. Confirmatory 3D fixes standard, contextual, and copyright items to response-process factors. Confirmatory 7D fixes items to HarmBench harm-domain factors. Lower is better in the fit columns. Bold marks the best value in each column. AIC and BIC are means across splits and are rounded.
  • DIF Flags: Under the single aggregate score, the authors found 13 to 17 items where OpenAI and Anthropic models performed differently despite having the same overall "ability." However, when models were matched on more specific 3D or 7D sub-scores, these flags almost entirely vanished (see Table 2). This confirms that the "developer difference" is often just an artifact of how we choose to group and average different behaviors.
Table 2: Family-wise DIF flags after matching models on the relevant ability estimate. Rows include the single score, 3D response-process scopes, and 7D HarmBench-domain scopes. MH is the Mantel-Haenszel screen. Logit is the ridge-logistic sensitivity screen. Cutoffs are the 95th percentile of the maximum statistic across 5,000 bootstrap samples simulated with no group differences present.

Limitations and Future Work

  • Convenience Sample: The study used a snapshot of 81 models available in HELM. As models evolve, the dimensional structure of these benchmarks may change.
  • Judge Bias: The "ground truth" for passes/fails came from LLM judges (GPT-4o and Llama-3), which may carry their own systematic biases into the response matrix.
  • Binarization: The authors binarized 5-point rubric scores into pass/fail, which might discard useful nuance in model behavior.
  • Future Directions: The authors suggest that benchmarks should "earn" their single-attribute reading through psychometric audits before they are used to rank models. Future work could apply these audits to other domains like reasoning or coding.

Summary updated

Abstract

Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models with similar overall scores can have very different attribute profiles. Comparing models is more tractable at the level of individual attributes, yet it is often unclear whether even a single dataset's scores isolate any single attribute. One plausible candidate for such an attribute is harmful refusal, a model's tendency to refuse dangerous or policy-violating prompts. We examine whether it constitutes a single, measurable attribute in HELM Safety. Using a construct validity framework that stipulates that an attribute must exist before a test can measure it, we start with HELM Safety's four datasets that might plausibly target harmful refusal, but find that three are saturated. We subject the remaining dataset, HarmBench, to two psychometric tests to determine if a single attribute like harmful refusal could stand behind its score. First, multidimensional item response theory modeling strongly suggests that HarmBench does not measure a singular attribute. Second, a differential item functioning analysis finds items where models from different developers with the same refusal ability score differently. These flags largely disappear under scope-specific matching, a pattern consistent with aggregation effects but not sufficient to rule out domain-specific developer differences. Zooming out, HarmBench collapses distinct harm behaviors into one score, and the overall HELM safety aggregate further collapses HarmBench and scores from other datasets into a single top-line number. Any safety score that averages over datasets and items can hide saturation and conflate behaviors this way. We argue that a score should earn its single-attribute reading before models are compared with it.
BibTeX
@misc{stewart2026searching,
  author = {Christopher M. Stewart and Preston Botter and Natalie Sarabosing and Muye Zhang and Rachel Phinnemore and Shalini Ghosh and Hong Shen and Hoda Heidari},
  title = {{Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark}},
  year = {2026},
  eprint = {2610.12409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.12409},
}
Figure