Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control

Yonghui Yang, Wenjian Tao, Jilong Liu, Xingyu Zhu, Junfeng Fang, Weibiao Huang, Le Wu, Richang Hong, Tat-Sent Chua

TL;DR

Current safety alignment techniques for Large Language Models (LLMs) are highly sensitive to noisy data and out-of-distribution prompts due to sharp, fragile optimization landscapes. This paper introduces ShaPO (Sharpness-aware Preference Optimization), a framework that selectively smooths the loss landscape specifically within a small, safety-critical parameter subspace. By avoiding uniform regularization, ShaPO significantly boosts safety robustness under domain shifts and noisy supervision without sacrificing general model performance.

What Problem Does This Paper Address?

Aligning LLMs with human safety preferences typically relies on preference-based optimization methods like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). While effective on curated training benchmarks, these aligned models often remain brittle in deployment. When faced with malicious out-of-distribution (OOD) prompts or trained on noisy, mislabeled preference data, their safety guardrails easily crumble.

Prior attempts to fix this have been overwhelmingly data-centric—focusing on filtering data, reweighting samples, or applying distributionally robust optimization. The authors argue that this overlooks a fundamental issue: optimization-induced fragility. Standard preference optimization often results in sharp loss landscapes where minor parameter perturbations lead to massive safety failures. While tools like Sharpness-Aware Minimization (SAM) can smooth the landscape, applying them uniformly across billions of parameters over-regularizes the model, stripping away its general capabilities and harming OOD generalization.

Methods

The authors propose ShaPO (Sharpness-aware Preference Optimization), which restricts geometry control exclusively to an alignment-critical parameter subspace.

The framework operates in three main steps:

  1. Safety-Critical Subspace Approximation: The authors train a lightweight linear probe on the residual stream of the LLM's final layer to distinguish between safe and unsafe responses. They then calculate the cosine similarity between this probe's directional weights and the internal value vectors of the transformer blocks. The top-performing neurons (conservatively set to the top 1%) define the safety-critical subspace $\mathcal{S}$.
  2. Selective Geometry Control: Instead of perturbing all weights during training, ShaPO applies SAM-style adversarial perturbations only to the parameters within $\mathcal{S}$, forcing the optimization process to find a flatter, more robust minimum for safety behaviors.
  3. Multi-Level Instantiations: ShaPO is implemented at two distinct levels:
  4. Token-level ShaPO: Directly stabilizes likelihood-based surrogate objectives like standard DPO without needing an external reward model.
  5. Reward-level ShaPO: Uses a pretrained reward model to provide soft preference targets, anchoring the geometry control to semantic safety signals rather than raw token probabilities.
Figure 2. Overview of ShaPO , a robust preference optimization framework with selective geometry control. ShaPO minimizes the worst-case alignment loss under adversarial perturbations restricted to the alignment-critical parameter subspace. Here is the instantiation of ShaPO at reward level.

Datasets

The models were trained and evaluated using several prominent safety benchmarks:

  • Training: PKU-SafeRLHF-30K (a large-scale dataset containing pairwise human preferences for safety and helpfulness).
  • Evaluation (In-Distribution): Held-out validation set of PKU-SafeRLHF-30K.
  • Evaluation (Out-of-Distribution / Domain Shift):
  • HH-RLHF-Safety: Focuses on the helpfulness vs. harmlessness trade-off.
  • Do-Not-Answer: Evaluates precision in refusing explicitly unsafe requests.
  • HarmBench: Comprises adversarially constructed prompts designed to induce harm.
  • SaladBench: Evaluates complex, compositional, and obfuscated harms.

Key Insights and Ideas

  • Safety is Structurally Localized: An empirical analysis revealed that perturbing fewer than 0.5% of probe-identified safety neurons accounts for over 80% of the worst-case alignment loss increase. This concentration justifies why selective geometry control works so well without hurting non-safety parameters.
Figure 1. Cumulative contribution to worst-case alignment loss under parameter perturbations. We compare the fraction of the total worst-case loss increase accounted for by perturbing probe- identified safety-critical neurons (Top-K) versus randomly selected neurons of the same size (Random-K).
  • Uniform Regularization Hurts OOD Performance: Uniformly smoothing the entire model suppresses domain-adaptive components essential for handling distributional variation. Selective control strikes a superior balance between sharpness reduction and alignment preservation.
  • Composability with Data-Centric Methods: Because ShaPO operates on the parameter/optimization axis rather than the data axis, it is fully orthogonal to data-centric robust methods (like Dr.DPO). Combining them yields additive robustness gains.

Key Results

The authors evaluated ShaPO across multiple model families and scales (Pythia-2.8B, LLaMA-3.2-3B, LLaMA-3-8B, Qwen2.5-7B).

  • Robustness under Domain Shift: ShaPO-T (token-level) and ShaPO-R (reward-level) consistently achieved the lowest average Attack Success Rate (ASR) across all OOD benchmarks. For instance, on LLaMA-3-8B, ShaPO-R brought the average ASR down to 0.70%, compared to DPO's 22.85% and Dr.DPO's 2.09%.
  • Resilience to Noisy Supervision: In experiments where up to 40% of the preference labels were intentionally flipped during training, ShaPO-R maintained a remarkably stable win rate, significantly outperforming standard DPO and data-robust baselines which degraded sharply under high noise.
Figure 3. Sensitivities to noise environments, we compare the win rate of all methods across different preference data flipping ratios (10%, 20%, 40%). The left is the results on Pythia-2.8B, and the right is on LLaMA-3.2-3B backbone.
  • Flatter Landscapes: Quantitative analysis confirmed that ShaPO-T successfully minimized the maximum Hessian eigenvalue (Max HE) on LLaMA-3-8B from 14.95 (DPO) down to 6.24, proving a significantly flatter loss landscape.

Limitations and Future Work

  • Computational Overhead: ShaPO requires a two-pass forward-backward optimization scheme inherent to SAM-style updates. Although the authors mitigate this by performing the sharpness update once every $T$ steps instead of every iteration, it still introduces additional training overhead.
  • Approximation Dependency: Identifying the safety-critical subspace relies on approximate signals from a linear probe. While coarse geometric cues proved sufficient here, more sophisticated mechanistic interpretability techniques could yield even cleaner subspace boundaries.
  • Scalability: Future work is required to validate the efficiency and scalability of selective geometry control on much larger model scales (e.g., 70B+ parameters).

Summary updated

Abstract

Safety alignment of large language models remains brittle under domain shift and noisy preference supervision. Most existing robust alignment methods focus on uncertainty in alignment data, while overlooking optimization-induced fragility in preference-based objectives. In this work, we revisit robustness for LLM safety alignment from an optimization geometry perspective, and argue that robustness failures cannot be addressed by data-centric methods alone. We propose \textit{ShaPO}, a geometry-aware preference optimization framework that enforces worst-case alignment objectives via selective geometry control over alignment-critical parameter subspace. By avoiding uniform geometry constraints, ShaPO mitigates the over-regularization that can harm robustness under distribution shift. We instantiate ShaPO at two levels: token-level ShaPO stabilizes likelihood-based surrogate optimization, while reward-level ShaPO enforces reward-consistent optimization under noisy supervision. Across diverse safety benchmarks and noisy preference settings, ShaPO consistently improves safety robustness over popular preference optimization methods. Moreover, ShaPO composes cleanly with data-robust objectives, yielding additional gains and empirically supporting the proposed optimization-geometry perspective. The code is available at https://github.com/liujilong0116/ShaPO.
BibTeX
@inproceedings{yang2026revisiting,
  author = {Yonghui Yang and Wenjian Tao and Jilong Liu and Xingyu Zhu and Junfeng Fang and Weibiao Huang and Le Wu and Richang Hong and Tat-Sent Chua},
  title = {{Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=SZ9ePNI4Ps},
}
Figure