TL;DR
This paper investigates whether common AI safety benchmarks truly measure progress in AI safety or if they are merely proxies for general model capabilities and training compute. The authors find that many safety benchmarks are highly correlated with these capabilities, leading to "safetywashing" where improvements in general AI performance are misconstrued as safety advancements. This work proposes a more rigorous, empirically grounded approach to defining and measuring AI safety.
What Problem Does This Paper Address?
The rapid advancement of AI systems has spurred significant interest in "AI safety" research, aiming to mitigate potential risks. However, the field of AI safety is currently ill-defined and its progress is inconsistently measured. A key challenge is distinguishing genuine safety improvements from advancements in general AI capabilities (like reasoning and knowledge). This ambiguity creates a risk of "safetywashing," where improvements in general AI performance are presented as safety progress, potentially misleading researchers and the public. The paper aims to provide empirical evidence to clarify this distinction and propose a more robust framework for AI safety research.
Methods
The core methodology involves an empirical meta-analysis of AI safety benchmarks. The authors quantify the correlation between safety benchmark scores and two key factors:
- General Capabilities: They use Principal Component Analysis (PCA) on a set of established capabilities benchmarks (like MMLU, GSM8K) to derive a "capabilities score" for each model. This score represents the direction along which models vary the most in terms of general capabilities.
- Training Compute: They collect or estimate the training compute (FLOPs) for the models.
By computing Spearman correlations between safety benchmark scores and these two factors across numerous models, they empirically assess whether a safety benchmark is measuring distinct safety properties or is primarily driven by general capabilities and compute.
Datasets
The paper utilizes a wide array of existing datasets for evaluating AI models across various safety and capability dimensions. These include:
- Capabilities Benchmarks: LogiQA, PIQA, Hellaswag, Winogrande, COPA, MedQA, ARC Challenge, MMLU, MATH, LAMBADA, GSM8K, BBH.
- Alignment Benchmarks: LMSYS Chatbot Arena, MT-Bench.
- Machine Ethics Benchmarks: ETHICS, MACHIAVELLI, Sycophancy.
- Bias Benchmarks: BBQ, CrowS-Pairs, Discrim-Eval.
- Truthfulness Benchmarks: TruthfulQA (MC1 and Generation), HaluEval.
- Scalable Oversight Benchmarks: GPQA, QUALITY.
- Adversarial Robustness Benchmarks: AdvGLUE, AdvGLUE++, Human Jailbreaks, TAP, GCG, ImageNet-A, PGD on ImageNet.
- Weaponization Capabilities Benchmarks: WMDP.
- Strict Instruction Following Benchmarks: IFEval, RuLES.
- Calibration Benchmarks: Brier Score, RMSCE (using MMLU, PIQA, MedQA datasets).
The paper does not introduce new benchmarks but rather analyzes existing ones.
Key Insights and Ideas
- The "Safetywashing" Phenomenon: The most striking insight is the strong correlation between many AI safety benchmarks and general model capabilities/compute. This suggests that simply scaling up models or improving general performance can lead to better scores on these "safety" benchmarks, making it difficult to claim genuine safety progress.
- Empirical Evidence Over Intuition: The paper emphasizes the inadequacy of relying solely on intuitive arguments to define AI safety research areas. Empirical measurement of correlations is crucial for understanding what benchmarks actually capture.
- Differential Safety Progress is Key: True AI safety progress requires improvements that are decorrelated from general capabilities. Benchmarks that are highly correlated with capabilities are less likely to drive this differential progress.
- Alignment Theory's Limitations: The authors suggest that alignment theory, while influential, can be a "counterproductive paradigm" if it leads to research that prioritizes capabilities that happen to improve safety benchmarks, rather than addressing fundamental safety concerns.
- Nuance Across Safety Areas: Not all safety areas are equally prone to safetywashing. Bias, dynamic adversarial robustness, and calibration show lower correlations with capabilities, suggesting they might be more robust to this issue.
Key Results
The paper presents extensive empirical results across numerous models and benchmarks. Key findings include:
- High Correlations: Many safety benchmarks, particularly in areas like alignment, scalable oversight, and truthfulness, exhibit high correlations with capabilities (often >70-80%). For example, MT-Bench and LMSYS Chatbot Arena show correlations of 78.7% and 62.1% respectively with capabilities. TruthfulQA MC1 has an 81.2% correlation with capabilities.
- Compute as a Driver: Training compute is also a significant factor, often showing high correlations with safety benchmark scores, reinforcing the idea that scaling can drive performance on these metrics.
- Low Correlations in Some Areas: Benchmarks related to bias, dynamic adversarial robustness, and calibration show lower correlations. For instance, bias benchmarks like CrowS-Pairs English and Discrim-Eval have correlations of -17.5% and 2.8% respectively.
- Weaponization is Anticorrelated: Weaponization capabilities benchmarks (WMDP) show strong negative correlations with capabilities (-88.6% for chat models), indicating that as models become more capable, their potential for misuse increases.
- Calibration Nuance: The Brier Score for calibration is highly correlated with accuracy (95.5% for chat models), making it prone to safetywashing. However, RMS calibration error shows low correlation (20.1% for chat models), suggesting it measures a more distinct property.
Limitations and Future Work
- Limitations:
- The paper acknowledges that some information on training compute is not publicly available, leading to estimations.
- The analysis focuses on existing benchmarks and does not introduce new ones.
- The definition of "safety" itself is complex and can evolve.
- Future Work:
- The authors propose developing more meaningful safety metrics that are empirically separable from general capabilities.
- They advocate for future safety benchmarks to report their correlation with upstream model capabilities.
- The research suggests a need for a more rigorous, empirically driven framework for AI safety research, moving beyond purely theoretical or intuitive arguments.
- Further investigation into how to design benchmarks that incentivize "differential safety progress" is recommended.
Summary updated