Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?

Richard Ren, Steven Basart, Adam Khoja, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Alexander Pan, Gabriel Mukobi, Ryan H. Kim, Stephen Fitz, Dan Hendrycks

TL;DR

This paper demonstrates that many popular AI safety benchmarks are essentially proxies for general model capabilities and training compute. By showing that "safety" scores often improve simply by scaling up models, the authors warn against "safetywashing"—the practice of misrepresenting general capability gains as targeted safety progress—and call for benchmarks that are empirically decorrelated from raw intelligence.

What Problem Does This Paper Address?

As AI models become more powerful, the research community has scrambled to create benchmarks that measure "safety" properties like alignment, truthfulness, and robustness. However, the field lacks a rigorous definition of what makes a research goal "safety-related" versus just "better engineering."

The authors identify a major risk: safetywashing. If a safety benchmark is highly correlated with general capabilities (like reasoning or knowledge), a developer can claim safety progress simply by training a larger model with more data. This obscures whether we are actually making models safer or just making them smarter—which, as the paper notes, can sometimes make them more dangerous in specific contexts (e.g., weaponization knowledge).

Methods

The authors propose an empirical framework to test if a safety benchmark is actually measuring something distinct from general intelligence. Their methodology follows three main steps:

  1. Establish a Capabilities Baseline: They collect scores from dozens of models across standard capability benchmarks (e.g., MMLU, GSM8K).
  2. Extract a "Capabilities Score": Using Principal Component Analysis (PCA), they identify the first principal component (PC1), which explains ~70% of the variance across all capability tasks. This serves as a proxy for a model's "general intelligence."
  3. Correlation Analysis: They compute the Spearman correlation between this Capabilities Score and various safety benchmarks. They also correlate safety scores with raw training compute (FLOPs).
Figure 3: Step 1: We produce a matrix of scores for a set of language models evaluated on a set of capabilities and safety benchmarks. Step 2: We extract the first principal component of the capabilities benchmarks and use it to compute a capabilities score for each model. Step 3: We identify whether safety benchmarks have high capabilities correlations using Spearman’s correlation.

A high correlation suggests the benchmark is "liable for safetywashing," while a low or negative correlation suggests it measures a distinct safety property.

Datasets

The study performs a meta-analysis across a wide array of existing datasets:

  • Capabilities: MMLU, GSM8K, Hellaswag, PIQA, ARC, and others.
  • Alignment: MT-Bench, LMSYS Chatbot Arena.
  • Machine Ethics: ETHICS, MACHIAVELLI, Sycophancy.
  • Bias: BBQ, CrowS-Pairs, Discrim-Eval.
  • Truthfulness & Oversight: TruthfulQA, GPQA, QuALITY.
  • Security: ANLI, AdvGLUE (Adversarial Robustness), and WMDP (Weaponization).

The analysis spans 27 base models and 26 chat-tuned models, including the Llama, Mistral, and Qwen families.

Key Insights and Ideas

  • The Jangle Fallacy: The authors highlight the "jangle fallacy"—the mistaken belief that two constructs are different just because they have different names. Many "safety" tasks are just "capabilities" tasks with a different label.
  • Alignment is mostly Scaling: Popular alignment benchmarks like MT-Bench correlate at 78.7% with general capabilities. This suggests that "aligning" a model often just means making it a better, more helpful assistant through scale.
  • Safety vs. Risk: Improving capabilities might improve "safety" scores (like truthfulness) while simultaneously increasing "risks" (like the ability to help create bioweapons).
  • Metric Choice Matters: In calibration research, the Brier Score is highly correlated with accuracy (95.5%), making it a poor safety metric. However, RMS Calibration Error shows low correlation (20.1%), making it a much better tool for measuring a model's actual uncertainty independent of its intelligence.
A visual representation of how safety benchmarks can be categorized by their liability to safetywashing based on their correlation with compute.

Key Results

The paper categorizes safety areas based on their entanglement with capabilities:

  • High Correlation (Prone to Safetywashing): Alignment (MT-Bench: 78.7%), Truthfulness (TruthfulQA: 81.2%), and Scalable Oversight (QuALITY: 88.8%).
  • Low Correlation (Likely Distinct): Bias (BBQ Ambiguous: -37.3%), Calibration (RMSCE: 20.1%), and dynamic Adversarial Robustness (Jailbreaks: ~ -30%).
  • Negative Correlation (Risk Increases with Scale): Weaponization (WMDP: -87.5%) and Sycophancy (-66.8%). As models get smarter, they gain more hazardous knowledge and become better at telling users what they want to hear rather than the truth.
Figure 2 illustrates how a "safety technique" might be credited for progress that is actually just following the default trajectory of capability improvements.

Limitations and Future Work

The authors acknowledge that their study is limited to well-established benchmarks and does not cover emerging areas like transparency, anomaly detection, or trojan discovery. Furthermore, a low correlation doesn't automatically mean a benchmark is good; it just means it isn't measuring general capabilities.

Future Work: The authors recommend that all new safety benchmarks empirically report their correlation with upstream capabilities. They urge the community to design "decorrelated benchmarks" that implicitly control for scale, ensuring that safety research drives progress that wouldn't have happened simply by building a bigger model.

Summary updated

Abstract

As artificial intelligence systems grow more powerful, there has been increasing interest in "AI safety" research to address emerging and future risks. However, the field of AI safety remains poorly defined and inconsistently measured, leading to confusion about how researchers can contribute. This lack of clarity is compounded by the unclear relationship between AI safety benchmarks and upstream general capabilities (e.g., general knowledge and reasoning). To address these issues, we conduct a comprehensive meta-analysis of AI safety benchmarks, empirically analyzing their correlation with general capabilities across dozens of models and providing a survey of existing directions in AI safety. Our findings reveal that many safety benchmarks highly correlate with both upstream model capabilities and training compute, potentially enabling "safetywashing"--where capability improvements are misrepresented as safety advancements. Based on these findings, we propose an empirical foundation for developing more meaningful safety metrics and define AI safety in a machine learning research context as a set of clearly delineated research goals that are empirically separable from generic capabilities advancements. In doing so, we aim to provide a more rigorous framework for AI safety research, advancing the science of safety evaluations and clarifying the path towards measurable progress.
BibTeX
@inproceedings{ren2024safetywashing,
  author = {Richard Ren and Steven Basart and Adam Khoja and Alice Gatti and Long Phan and Xuwang Yin and Mantas Mazeika and Alexander Pan and Gabriel Mukobi and Ryan H. Kim and Stephen Fitz and Dan Hendrycks},
  title = {{Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=pvtX1GbLL2},
}
Figure