Abstract
Supervised Fine-Tuning (SFT) is the standard for adapting large language models (LLMs) to specific tasks, yet it frequently results in catastrophic forgetting of pre-trained knowledge and brittle safety guardrails. We present Learning-from-the-Undesirable (LfU), a regularization framework that enhances model robustness by maintaining stable internal representations even when the model is intentionally steered toward undesirable behaviors. By simulating "worst-case" updates via gradient ascent and enforcing consistency in hidden states, LfU significantly improves out-of-distribution generalization, prompt robustness, and resistance to adversarial safety degradation.
Introduction: The Fragility of Supervised Fine-Tuning
Adapting a pre-trained language model to a specific task usually involves Supervised Fine-Tuning (SFT). However, SFT often has a "narrowing" effect. When training data is limited or highly specific, the model frequently overfits to superficial patterns rather than learning robust reasoning. This leads to three major issues:
- Catastrophic Forgetting: The model loses the broad world knowledge and general capabilities gained during pre-training.
- Instability: Performance becomes highly sensitive to prompt wording (e.g., "Solve this" vs. "Let's solve this").
- Fragile Alignment: Safety guardrails are often superficial. Adversarial fine-tuning can quickly undo a model's safety training, making it susceptible to generating harmful content.
Methods: Learning from the Undesirable
The authors propose LfU, which utilizes a novel form of consistency regularization. Instead of merely training the model to predict the next token, LfU forces the model's internal hidden states to remain consistent even when the model is intentionally perturbed toward "undesirable" behavior.
The "Undesirable" Step
The core innovation is the creation of a controlled perturbation. Unlike methods that add random noise (such as NEFTune), LfU simulates an "undesirable" update by creating an auxiliary version of the model and performing a single step of gradient ascent on the SFT loss. This intentionally steers the model away from the target behavior, identifying the directions in parameter space most prone to overfitting or forgetting.
Consistency Regularization
The training objective minimizes the Mean Squared Error (MSE) between the internal representations (hidden states) of the original model and this "undesirable" auxiliary model. By forcing these representations to stay close, the model learns to be resilient against updates that would otherwise cause it to collapse into narrow, overfitted minima.
The authors offer two implementations: LfU (LoRA), which uses Low-Rank Adaptation parameters, and LfU (RepS), a more efficient version using "representation steering" vectors that is nearly twice as fast.
Key Results and Evaluation
LfU was evaluated across a massive battery of benchmarks, including GSM8k (math), ARC-Challenge (science), and multi-task datasets like Alpagasus Dolly 3k.
Performance and Robustness
On Llama-3.1-8B, LfU improved math task performance by 16.8% compared to vanilla SFT. Furthermore, it dramatically improved prompt robustness, reducing the standard deviation of performance across different prompt variations by 92.1%. This suggests that LfU helps the model learn underlying logic rather than brittle pattern matching.
Safety Resilience
A critical finding is LfU's resistance to adversarial "un-alignment." In scenarios where a model is intentionally fine-tuned to be harmful, LfU proved remarkably stubborn. On the HEx-PHI safety benchmark, vanilla SFT's Attack Success Rate (ASR) surged to 34.8% after just five adversarial steps. In contrast, LfU maintained a low ASR of 5.8%, demonstrating that its safety guardrails are deeply integrated into its internal representations.
Conclusion
LfU introduces a powerful paradigm for model adaptation: by explicitly learning to ignore "undesirable" updates, models become more stable, generalizable, and safe. The method's success in maintaining internal representation stability suggests that focusing on hidden states rather than just output logits is key to preventing catastrophic forgetting in the next generation of language models.
Summary updated