Abstract
The emergence of finetuning-as-a-service has revealed a new vulnerability in large language models (LLMs). A mere handful of malicious data uploaded by users can subtly manipulate the finetuning process, resulting in an alignment-broken model. Existing methods to counteract fine-tuning attacks typically require substantial computational resources. Even with parameter-efficient techniques like LoRA, gradient updates remain essential. To address these challenges, we propose Neuron-Level Safety Realignment (NLSR), a training-free framework that restores the safety of LLMs based on the similarity difference of safety-critical neurons before and after fine-tuning. The core of our framework is first to construct a safety reference model from an initially aligned model to amplify safety-related features in neurons. We then utilize this reference model to identify safety-critical neurons, which we prepare as patches. Finally, we selectively restore only those neurons that exhibit significant similarity differences by transplanting these prepared patches, thereby minimally altering the fine-tuned model. Extensive experiments demonstrate significant safety enhancements in fine-tuned models across multiple downstream tasks, while greatly maintaining task-level accuracy.
Introduction
The widespread adoption of "fine-tuning-as-a-service" has introduced significant security risks. While base models are typically aligned to ensure safety, research indicates that fine-tuning on even a minimal amount of malicious data (as little as 1%) can compromise these safeguards, leading to "harmful fine-tuning attacks." Existing defenses often necessitate expensive retraining or impose constraints that hinder the model's ability to learn new tasks.
To bridge this gap, we introduce Neuron-Level Safety Realignment (NLSR). NLSR is a training-free framework designed to restore safety in compromised LLMs by surgically "patching" specific safety-critical neurons. By leveraging a super-aligned reference model, NLSR reduces harmful outputs while preserving performance on downstream tasks.
Methodology
The NLSR framework operates in three primary stages, as illustrated in Figure 2.
- Construction of a Safety Reference Model: Instead of relying solely on the original aligned model, NLSR employs "weak-to-strong extrapolation" to create a "super-aligned" reference. This involves amplifying safety-related weights within a LoRA (Low-Rank Adaptation) configuration to make safety-critical features more discernible.
- Recognition of Safety-Critical Neurons: Utilizing Singular Value Decomposition (SVD), the framework identifies neurons most responsible for safety behaviors, creating a spatial map of safety logic within the model's layers.
- Neuron-Level Correction: The system compares safety neurons of the fine-tuned model with those of the reference. If similarity is low—indicating corruption—healthy neurons are "transplanted" from the reference. This process uses probability-based layer pruning to ensure only compromised components are corrected.
Experimental Results and Insights
The effectiveness of NLSR was evaluated across benchmarks including SST-2, AGNEWS, and GSM8K, with safety measured via HarmBench and QA-Moderation.
A key finding is that harmful fine-tuning disrupts output patterns rather than erasing safety concepts. Hidden state analysis showed that "safe" vs. "unsafe" inputs could still be classified with over 95% accuracy. NLSR's surgical approach targets individual neurons, preserving task-specific knowledge while reverting only the parameters causing safety leaks.
As shown in Figure 5, the pre-amplification coefficient $\beta$ plays a crucial role. Increasing $\beta$ significantly reduces the harmfulness score (measured on BeaverTails) while maintaining or even slightly improving accuracy on downstream tasks like tinyMMLU and tinyHellaswag. This demonstrates that the "super-aligned" reference provides a clearer signal for identifying and restoring safety-critical neurons without degrading general performance.
Quantitatively, NLSR reduced the Harmfulness Score by 38.3% on SST-2 compared to an attacked model. It outperformed state-of-the-art methods like SafeLoRA, achieving a 30.3% lower harmfulness score and a 1.1% gain in task accuracy. The method proved robust across architectures including Llama3-8B, Qwen2-7B, and Mistral-7B.
Conclusion
NLSR offers a precise, training-free solution for realigning LLMs after harmful fine-tuning. By focusing on neuron-level restoration, it maintains a balance between safety and utility. Future work will address dependencies on initial alignment quality, scalability to full-parameter fine-tuning, and resilience against adaptive attacks.
Summary updated