Abstract
Large Language Models (LLMs) are increasingly deployed via "fine-tuning-as-a-service" (FTaaS), allowing users to customize models for specific tasks. However, this flexibility introduces a significant security risk: "harmful fine-tuning." Even a tiny fraction of malicious data (as low as 1%) can cause a model's safety guardrails to collapse. We introduce Neuron-Level Safety Realignment (NLSR), a training-free framework designed to surgically restore safety. NLSR identifies safety-critical neurons and replaces compromised ones with healthy versions from a super-aligned reference model. Experimental results demonstrate that NLSR significantly reduces harmfulness while maintaining near-perfect utility on downstream tasks.
Introduction
The widespread adoption of LLMs has necessitated robust safety alignment, typically achieved through Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). Despite these efforts, the rise of FTaaS has created a loophole. When users fine-tune an aligned model on custom datasets, they may inadvertently or intentionally trigger "alignment drift," where the model's safety constraints are weakened.
Existing defenses against such attacks often involve retraining the model or applying broad, layer-wise updates. These methods are computationally expensive and frequently degrade the model's performance on the target task. NLSR addresses these challenges by offering a surgical, training-free approach that operates at the individual neuron level.
Methodology: The NLSR Framework
The NLSR framework consists of three primary stages designed to identify and fix safety-broken components without requiring additional gradient steps.
1. Construction of a Safety Reference Model
To effectively identify safety-critical neurons, the framework first constructs a "super-aligned" reference model ($F_{w_e}$). This is achieved through LoRA extrapolation, which interpolates between a weakly aligned model and a strongly aligned one. This process amplifies safety-related features, making the "safety signal" within the weights more discernible.
2. Recognition of Safety-Critical Neurons
Once the reference model is established, NLSR identifies the specific neurons responsible for safety. By applying Singular Value Decomposition (SVD) and rank reduction to the LoRA weights, the system generates a "position mask" ($M$) that highlights safety-critical components. This allows the system to focus its restoration efforts only on the neurons that matter most for alignment.
3. Neuron-Level Restoration
The final stage involves a "transplant" of healthy neurons into the customized model ($F_{W_t}$). This stage includes:
- Similarity Check: Comparing the safety-critical neurons of the customized model against the super-aligned reference.
- Adaptive Layer Pruning: A probability-based selection process determines which layers require intervention. Layers that retain high similarity to the reference are left untouched to preserve task-specific utility.
- Neuron-Level Correction: For compromised layers, the "broken" neurons are replaced with "patches" derived from the reference model. This mathematical update to the LoRA weights is entirely training-free.
Experimental Evaluation
The authors evaluated NLSR using benchmarks such as PKU-SafeRLHF and BeaverTails for safety, and SST-2, AGNEWS, and GSM8K for utility.
Key Results
- Safety Restoration: In sentiment analysis tasks (SST-2), NLSR reduced the harmfulness score from 61.1% (in a standard aligned model) to 22.8%. This outperformed the previous state-of-the-art method, SafeLoRA, which achieved a score of 53.1%.
- Utility Preservation: Remarkably, NLSR maintained a fine-tuning accuracy of 95.1% on SST-2, showing virtually no performance loss compared to the compromised model.
- Robustness and Efficiency: The framework remained effective even when the poisoning ratio of the fine-tuning data reached 30%. Furthermore, because it is training-free, the realignment process occurs in seconds, making it highly suitable for real-time deployment in FTaaS pipelines.
Discussion and Conclusion
NLSR demonstrates that safety in LLMs is "regional" and can be managed with surgical precision. By focusing on individual neurons rather than entire layers, the framework avoids the "blunt instrument" problem of previous defenses.
While currently optimized for LoRA-based fine-tuning, future work could extend these principles to full-parameter updates. NLSR represents a significant step toward practical, efficient, and robust safety guardrails for the next generation of customizable AI models.
Summary updated