Abstract
Safety alignment of large language models (LLMs) has been gaining increasing attention. However, current safety-aligned LLMs suffer from fragile and imbalanced safety mechanisms, which can still be induced to generate unsafe responses, exhibit over-safety by rejecting safe user inputs, and fail to preserve general utility after safety alignment. To this end, we propose a novel post safety alignment (PSA) method to address these inherent and emerging safety challenges, including safety enhancement, over-safety mitigation, and utility preservation. In specific, we introduce SAFEPATCHING, a novel framework for comprehensive and efficient PSA, where two distinct safety patches are developed on harmful data to enhance safety and mitigate over-safety concerns, and then seamlessly integrated into the target LLM backbone without compromising its utility. Extensive experiments show that SAFEPATCHING achieves a more comprehensive and efficient PSA than baseline methods. It even enhances the utility of the backbone, further optimizing the balance between being helpful and harmless in current aligned LLMs. Also, SAFEPATCHING demonstrates its superiority in continual PSA scenarios.
Introduction: The Safety Trilemma
Safety alignment for Large Language Models (LLMs) is often a zero-sum game. Tightening safety to prevent jailbreaks often causes the model to become "over-safe," refusing harmless questions (e.g., "How do I kill a Python process?"). Conversely, aggressive safety tuning frequently degrades general intelligence—a phenomenon known as the "tax on alignment."
Researchers from the Harbin Institute of Technology and colleagues propose SAFEPATCHING, a novel framework designed to solve this "Safety Trilemma" by treating safety as a modular software patch rather than a full-system overhaul. This post-hoc alignment framework simultaneously enhances model safety, mitigates over-refusal, and preserves general utility.
The SAFEPATCHING Framework
SAFEPATCHING operates in two main stages: Patch Derivation and Controllable Patching. By deriving two distinct "safety patches" from harmful data and surgically integrating them into a backbone model using sparse parameter techniques, it achieves a superior balance between being helpful and harmless compared to traditional fine-tuning.
1. Patch Derivation (PD)
The framework derives two opposing patches from a set of harmful data ($D_h$):
- Safety Enhancement Patch ($P^{SE}$): Created using gradient ascent. This essentially tells the model to "unlearn" the probability of generating harmful responses.
- Over-Safety Mitigation Patch ($P^{OSM}$): Created using gradient descent on the same harmful data, but with a "malicious fine-tuning" objective. This forces the model to respond freely, effectively "erasing" inherent over-refusal mechanisms.
2. Controllable Patching (CP)
To avoid breaking the model's general utility, patches are not added blindly. The method employs Random Retention to sparsify the patches (e.g., keeping only 30% of parameters). Using SNIP scores (a metric for parameter importance), the framework identifies which neurons are most critical for safety vs. over-safety and prioritizes patching in the "difference set"—regions where the two objectives don't overlap—to prevent the patches from canceling each other out.
Spatial Separation of Safety Mechanisms
A key insight of this work is the discovery that safety and over-safety parameters are spatially separated within the model architecture.
As shown in the importance analysis, over-safety parameters are concentrated in the early attention layers, while safety-critical parameters are more prevalent in the mid-to-late Feed-Forward Network (FFN) layers. This separation allows for modular patching without significant interference between the two objectives.
Experimental Results
Evaluated on models like LLaMA-2-7B and LLaMA-3-8B across benchmarks including AdvBench (safety), XSTest (over-safety), and MMLU/MT-bench (utility), SAFEPATCHING demonstrated:
- Comprehensive Improvement: On LLaMA-2-7B, the jailbreak Attack Success Rate (ASR) dropped from 24% to 7.25%, while over-safety refusal on XSTest fell from 8% to 3.33%.
- Utility Preservation: Unlike baseline methods (like GA or NPO) which saw utility scores drop, SAFEPATCHING maintained or slightly exceeded the original backbone's performance. This is partly due to the removal of "preachy" over-refusals in harmless tasks.
- Inference Efficiency: Because the patches are merged directly into the weights, there is zero additional inference overhead, unlike decoding-based defenses which can slow down generation significantly.
Conclusion
SAFEPATCHING offers a modular, efficient approach to LLM safety. By treating safety as a "delta" and leveraging the spatial separation of safety mechanisms, it achieves a superior balance of being helpful and harmless. Future work will explore scaling these findings to 70B+ parameter models and validating domain-specific safety applications.
Summary updated