TL;DR
Current safety alignment techniques for Large Language Models (LLMs) are highly sensitive to noisy data and out-of-distribution prompts due to sharp, fragile optimization landscapes. This paper introduces ShaPO (Sharpness-aware Preference Optimization), a framework that selectively smooths the loss landscape specifically within a small, safety-critical parameter subspace. By avoiding uniform regularization, ShaPO significantly boosts safety robustness under domain shifts and noisy supervision without sacrificing general model performance.
What Problem Does This Paper Address?
Aligning LLMs with human safety preferences typically relies on preference-based optimization methods like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). While effective on curated training benchmarks, these aligned models often remain brittle in deployment. When faced with malicious out-of-distribution (OOD) prompts or trained on noisy, mislabeled preference data, their safety guardrails easily crumble.
Prior attempts to fix this have been overwhelmingly data-centric—focusing on filtering data, reweighting samples, or applying distributionally robust optimization. The authors argue that this overlooks a fundamental issue: optimization-induced fragility. Standard preference optimization often results in sharp loss landscapes where minor parameter perturbations lead to massive safety failures. While tools like Sharpness-Aware Minimization (SAM) can smooth the landscape, applying them uniformly across billions of parameters over-regularizes the model, stripping away its general capabilities and harming OOD generalization.
Methods
The authors propose ShaPO (Sharpness-aware Preference Optimization), which restricts geometry control exclusively to an alignment-critical parameter subspace.
The framework operates in three main steps:
- Safety-Critical Subspace Approximation: The authors train a lightweight linear probe on the residual stream of the LLM's final layer to distinguish between safe and unsafe responses. They then calculate the cosine similarity between this probe's directional weights and the internal value vectors of the transformer blocks. The top-performing neurons (conservatively set to the top 1%) define the safety-critical subspace $\mathcal{S}$.
- Selective Geometry Control: Instead of perturbing all weights during training, ShaPO applies SAM-style adversarial perturbations only to the parameters within $\mathcal{S}$, forcing the optimization process to find a flatter, more robust minimum for safety behaviors.
- Multi-Level Instantiations: ShaPO is implemented at two distinct levels:
- Token-level ShaPO: Directly stabilizes likelihood-based surrogate objectives like standard DPO without needing an external reward model.
- Reward-level ShaPO: Uses a pretrained reward model to provide soft preference targets, anchoring the geometry control to semantic safety signals rather than raw token probabilities.
Datasets
The models were trained and evaluated using several prominent safety benchmarks:
- Training: PKU-SafeRLHF-30K (a large-scale dataset containing pairwise human preferences for safety and helpfulness).
- Evaluation (In-Distribution): Held-out validation set of PKU-SafeRLHF-30K.
- Evaluation (Out-of-Distribution / Domain Shift):
- HH-RLHF-Safety: Focuses on the helpfulness vs. harmlessness trade-off.
- Do-Not-Answer: Evaluates precision in refusing explicitly unsafe requests.
- HarmBench: Comprises adversarially constructed prompts designed to induce harm.
- SaladBench: Evaluates complex, compositional, and obfuscated harms.
Key Insights and Ideas
- Safety is Structurally Localized: An empirical analysis revealed that perturbing fewer than 0.5% of probe-identified safety neurons accounts for over 80% of the worst-case alignment loss increase. This concentration justifies why selective geometry control works so well without hurting non-safety parameters.
- Uniform Regularization Hurts OOD Performance: Uniformly smoothing the entire model suppresses domain-adaptive components essential for handling distributional variation. Selective control strikes a superior balance between sharpness reduction and alignment preservation.
- Composability with Data-Centric Methods: Because ShaPO operates on the parameter/optimization axis rather than the data axis, it is fully orthogonal to data-centric robust methods (like Dr.DPO). Combining them yields additive robustness gains.
Key Results
The authors evaluated ShaPO across multiple model families and scales (Pythia-2.8B, LLaMA-3.2-3B, LLaMA-3-8B, Qwen2.5-7B).
- Robustness under Domain Shift: ShaPO-T (token-level) and ShaPO-R (reward-level) consistently achieved the lowest average Attack Success Rate (ASR) across all OOD benchmarks. For instance, on LLaMA-3-8B, ShaPO-R brought the average ASR down to 0.70%, compared to DPO's 22.85% and Dr.DPO's 2.09%.
- Resilience to Noisy Supervision: In experiments where up to 40% of the preference labels were intentionally flipped during training, ShaPO-R maintained a remarkably stable win rate, significantly outperforming standard DPO and data-robust baselines which degraded sharply under high noise.
- Flatter Landscapes: Quantitative analysis confirmed that ShaPO-T successfully minimized the maximum Hessian eigenvalue (Max HE) on LLaMA-3-8B from 14.95 (DPO) down to 6.24, proving a significantly flatter loss landscape.
Limitations and Future Work
- Computational Overhead: ShaPO requires a two-pass forward-backward optimization scheme inherent to SAM-style updates. Although the authors mitigate this by performing the sharpness update once every $T$ steps instead of every iteration, it still introduces additional training overhead.
- Approximation Dependency: Identifying the safety-critical subspace relies on approximate signals from a linear probe. While coarse geometric cues proved sufficient here, more sophisticated mechanistic interpretability techniques could yield even cleaner subspace boundaries.
- Scalability: Future work is required to validate the efficiency and scalability of selective geometry control on much larger model scales (e.g., 70B+ parameters).
Summary updated