TL;DR
This paper demonstrates that many popular AI safety benchmarks are essentially proxies for general model capabilities and training compute. By showing that "safety" scores often improve simply by scaling up models, the authors warn against "safetywashing"—the practice of misrepresenting general capability gains as targeted safety progress—and call for benchmarks that are empirically decorrelated from raw intelligence.
What Problem Does This Paper Address?
As AI models become more powerful, the research community has scrambled to create benchmarks that measure "safety" properties like alignment, truthfulness, and robustness. However, the field lacks a rigorous definition of what makes a research goal "safety-related" versus just "better engineering."
The authors identify a major risk: safetywashing. If a safety benchmark is highly correlated with general capabilities (like reasoning or knowledge), a developer can claim safety progress simply by training a larger model with more data. This obscures whether we are actually making models safer or just making them smarter—which, as the paper notes, can sometimes make them more dangerous in specific contexts (e.g., weaponization knowledge).
Methods
The authors propose an empirical framework to test if a safety benchmark is actually measuring something distinct from general intelligence. Their methodology follows three main steps:
- Establish a Capabilities Baseline: They collect scores from dozens of models across standard capability benchmarks (e.g., MMLU, GSM8K).
- Extract a "Capabilities Score": Using Principal Component Analysis (PCA), they identify the first principal component (PC1), which explains ~70% of the variance across all capability tasks. This serves as a proxy for a model's "general intelligence."
- Correlation Analysis: They compute the Spearman correlation between this Capabilities Score and various safety benchmarks. They also correlate safety scores with raw training compute (FLOPs).
A high correlation suggests the benchmark is "liable for safetywashing," while a low or negative correlation suggests it measures a distinct safety property.
Datasets
The study performs a meta-analysis across a wide array of existing datasets:
- Capabilities: MMLU, GSM8K, Hellaswag, PIQA, ARC, and others.
- Alignment: MT-Bench, LMSYS Chatbot Arena.
- Machine Ethics: ETHICS, MACHIAVELLI, Sycophancy.
- Bias: BBQ, CrowS-Pairs, Discrim-Eval.
- Truthfulness & Oversight: TruthfulQA, GPQA, QuALITY.
- Security: ANLI, AdvGLUE (Adversarial Robustness), and WMDP (Weaponization).
The analysis spans 27 base models and 26 chat-tuned models, including the Llama, Mistral, and Qwen families.
Key Insights and Ideas
- The Jangle Fallacy: The authors highlight the "jangle fallacy"—the mistaken belief that two constructs are different just because they have different names. Many "safety" tasks are just "capabilities" tasks with a different label.
- Alignment is mostly Scaling: Popular alignment benchmarks like MT-Bench correlate at 78.7% with general capabilities. This suggests that "aligning" a model often just means making it a better, more helpful assistant through scale.
- Safety vs. Risk: Improving capabilities might improve "safety" scores (like truthfulness) while simultaneously increasing "risks" (like the ability to help create bioweapons).
- Metric Choice Matters: In calibration research, the Brier Score is highly correlated with accuracy (95.5%), making it a poor safety metric. However, RMS Calibration Error shows low correlation (20.1%), making it a much better tool for measuring a model's actual uncertainty independent of its intelligence.
Key Results
The paper categorizes safety areas based on their entanglement with capabilities:
- High Correlation (Prone to Safetywashing): Alignment (MT-Bench: 78.7%), Truthfulness (TruthfulQA: 81.2%), and Scalable Oversight (QuALITY: 88.8%).
- Low Correlation (Likely Distinct): Bias (BBQ Ambiguous: -37.3%), Calibration (RMSCE: 20.1%), and dynamic Adversarial Robustness (Jailbreaks: ~ -30%).
- Negative Correlation (Risk Increases with Scale): Weaponization (WMDP: -87.5%) and Sycophancy (-66.8%). As models get smarter, they gain more hazardous knowledge and become better at telling users what they want to hear rather than the truth.
Limitations and Future Work
The authors acknowledge that their study is limited to well-established benchmarks and does not cover emerging areas like transparency, anomaly detection, or trojan discovery. Furthermore, a low correlation doesn't automatically mean a benchmark is good; it just means it isn't measuring general capabilities.
Future Work: The authors recommend that all new safety benchmarks empirically report their correlation with upstream capabilities. They urge the community to design "decorrelated benchmarks" that implicitly control for scale, ensuring that safety research drives progress that wouldn't have happened simply by building a bigger model.
Summary updated