TL;DR
This paper applies rigorous psychometric auditing to AI safety benchmarks, revealing that "harmful refusal" is not a single, unified trait. By analyzing the HarmBench dataset, the authors demonstrate that aggregating diverse safety behaviors into a single top-line score masks critical differences in model performance and developer-specific failure patterns.
What Problem Does This Paper Address?
AI safety leaderboards often boil down complex model behaviors into a single "safety score." This practice relies on an unstated assumption of construct validity: that there is a single, measurable attribute (like "harmful refusal") driving a model's responses across all test items.
If this assumption is false—if "safety" is actually a collection of unrelated behaviors—then a single score is misleading. Two models could achieve the same score while having completely different safety profiles (e.g., one is great at refusing cybercrime prompts but terrible at copyright, while the other is the opposite). The authors set out to audit whether the "harmful refusal" construct in the popular HELM Safety benchmark actually holds up under statistical scrutiny.
Methods
The authors treat AI benchmarks as psychological tests and apply two key techniques from the field of psychometrics:
- Multidimensional Item Response Theory (MIRT): Standard benchmarks assume a 1D "ability" axis. MIRT allows for multiple axes (dimensions). The authors compared a simple 1D model against multidimensional models (e.g., a 3D model separating "standard," "contextual," and "copyright" prompts) to see which better predicted how models actually pass or fail items.
- Differential Item Functioning (DIF): This technique looks for "construct-irrelevant variance." If "harmful refusal" is the only thing being measured, then two models with the same overall score should have the same probability of passing any specific item, regardless of who built them. If OpenAI models consistently fail an item that Anthropic models of equal "ability" pass, that item is flagged for DIF, suggesting it's measuring something other than the intended construct.
Datasets
The study analyzed the HELM Safety (v1.17.0) model pool, which includes 81 models. They initially looked at four datasets targeting harmful refusal:
- HarmBench: (400 items) The primary focus of the study.
- SimpleSafetyTests, AnthropicRedTeam, and XSTest (harmful subset): These were found to be "saturated," meaning modern models pass them so consistently (92-94% pass rates) that they provide little signal for item-level psychometric analysis.
Key Insights and Ideas
- Safety is not one-dimensional: The data strongly rejects the idea that a single "refusal" trait explains model behavior. Models that are good at refusing direct harmful requests aren't necessarily the same ones good at refusing to generate copyrighted text.
- Copyright is an outlier: Copyright items separate cleanly from other harm domains. The authors argue this is because copyright refusal is driven more by memorization during training than by a general "refusal disposition." Lumping them together is like averaging a student's math and history scores into a single "intelligence" metric.
- The "Developer Signature": When using a single aggregate score, models from different developers (OpenAI vs. Anthropic) show significant differences in how they handle specific items. These differences mostly disappear when you match models on more specific sub-scores (like "cybercrime" or "harassment"), suggesting that developer-specific training "signatures" are being flattened and hidden by top-line averages.
Key Results
- Predictive Power: Multidimensional models significantly outperformed the standard 1D model. As shown in Table 1, moving from a 1D model to a 3D model (separating response processes) dropped the held-out log-loss from 0.470 to 0.258, a massive improvement in explaining model behavior.
- DIF Flags: Under the single aggregate score, the authors found 13 to 17 items where OpenAI and Anthropic models performed differently despite having the same overall "ability." However, when models were matched on more specific 3D or 7D sub-scores, these flags almost entirely vanished (see Table 2). This confirms that the "developer difference" is often just an artifact of how we choose to group and average different behaviors.
Limitations and Future Work
- Convenience Sample: The study used a snapshot of 81 models available in HELM. As models evolve, the dimensional structure of these benchmarks may change.
- Judge Bias: The "ground truth" for passes/fails came from LLM judges (GPT-4o and Llama-3), which may carry their own systematic biases into the response matrix.
- Binarization: The authors binarized 5-point rubric scores into pass/fail, which might discard useful nuance in model behavior.
- Future Directions: The authors suggest that benchmarks should "earn" their single-attribute reading through psychometric audits before they are used to rank models. Future work could apply these audits to other domains like reasoning or coding.
Summary updated