Learning from the Undesirable: Robust Adaptation of Language Models without Forgetting

Yunhun Nam, Jaehyung Kim, Jongheon JeongarXiv 2025 · DOI · arXiv

Abstract

Supervised Fine-Tuning (SFT) is the standard for adapting large language models (LLMs) to specific tasks, yet it frequently results in catastrophic forgetting of pre-trained knowledge and brittle safety guardrails. We present Learning-from-the-Undesirable (LfU), a regularization framework that enhances model robustness by maintaining stable internal representations even when the model is intentionally steered toward undesirable behaviors. By simulating "worst-case" updates via gradient ascent and enforcing consistency in hidden states, LfU significantly improves out-of-distribution generalization, prompt robustness, and resistance to adversarial safety degradation.

Introduction: The Fragility of Supervised Fine-Tuning

Adapting a pre-trained language model to a specific task usually involves Supervised Fine-Tuning (SFT). However, SFT often has a "narrowing" effect. When training data is limited or highly specific, the model frequently overfits to superficial patterns rather than learning robust reasoning. This leads to three major issues:

  1. Catastrophic Forgetting: The model loses the broad world knowledge and general capabilities gained during pre-training.
  2. Instability: Performance becomes highly sensitive to prompt wording (e.g., "Solve this" vs. "Let's solve this").
  3. Fragile Alignment: Safety guardrails are often superficial. Adversarial fine-tuning can quickly undo a model's safety training, making it susceptible to generating harmful content.
Figure 1: Comparison of SFT and LfU performance across different tasks. SFT on a specific task (Task A) can lead to incorrect reasoning and the loss of general knowledge (Task B), while LfU maintains accuracy and safety across both.

Methods: Learning from the Undesirable

The authors propose LfU, which utilizes a novel form of consistency regularization. Instead of merely training the model to predict the next token, LfU forces the model's internal hidden states to remain consistent even when the model is intentionally perturbed toward "undesirable" behavior.

The "Undesirable" Step

The core innovation is the creation of a controlled perturbation. Unlike methods that add random noise (such as NEFTune), LfU simulates an "undesirable" update by creating an auxiliary version of the model and performing a single step of gradient ascent on the SFT loss. This intentionally steers the model away from the target behavior, identifying the directions in parameter space most prone to overfitting or forgetting.

Consistency Regularization

The training objective minimizes the Mean Squared Error (MSE) between the internal representations (hidden states) of the original model and this "undesirable" auxiliary model. By forcing these representations to stay close, the model learns to be resilient against updates that would otherwise cause it to collapse into narrow, overfitted minima.

The authors offer two implementations: LfU (LoRA), which uses Low-Rank Adaptation parameters, and LfU (RepS), a more efficient version using "representation steering" vectors that is nearly twice as fast.

Key Results and Evaluation

LfU was evaluated across a massive battery of benchmarks, including GSM8k (math), ARC-Challenge (science), and multi-task datasets like Alpagasus Dolly 3k.

Performance and Robustness

On Llama-3.1-8B, LfU improved math task performance by 16.8% compared to vanilla SFT. Furthermore, it dramatically improved prompt robustness, reducing the standard deviation of performance across different prompt variations by 92.1%. This suggests that LfU helps the model learn underlying logic rather than brittle pattern matching.

Safety Resilience

A critical finding is LfU's resistance to adversarial "un-alignment." In scenarios where a model is intentionally fine-tuned to be harmful, LfU proved remarkably stubborn. On the HEx-PHI safety benchmark, vanilla SFT's Attack Success Rate (ASR) surged to 34.8% after just five adversarial steps. In contrast, LfU maintained a low ASR of 5.8%, demonstrating that its safety guardrails are deeply integrated into its internal representations.

Figure 6: Attack Success Rate (ASR) over multiple adversarial fine-tuning steps. LfU demonstrates significantly higher resilience to safety degradation compared to standard SFT across different benchmarks.

Conclusion

LfU introduces a powerful paradigm for model adaptation: by explicitly learning to ignore "undesirable" updates, models become more stable, generalizable, and safe. The method's success in maintaining internal representation stability suggests that focusing on hidden states rather than just output logits is key to preventing catastrophic forgetting in the next generation of language models.

Summary updated

Abstract

Language models (LMs) are often adapted through supervised fine-tuning (SFT) to specialize their capabilities for downstream tasks. However, in typical scenarios where the fine-tuning data is limited, e.g., compared to pre-training, SFT can lead LMs to overfit, causing them to rely on spurious patterns within the target task or to compromise other broadly useful capabilities as a side effect of narrow specialization. In this paper, we propose Learning-from-the-Undesirable (LfU), a simple yet effective regularization scheme for SFT to mitigate overfitting issues when fine-tuning LMs with limited data. Specifically, we aim to regularize the fine-tuning process to favor solutions that are resilient to "undesirable" model updates, e.g., gradient ascent steps that steer the model toward undesirable behaviors. To this end, we propose a novel form of consistency regularization that directly aligns internal representations of the model with those after an undesirable update. By leveraging representation-level data augmentation through undesirable updates, LfU effectively promotes generalization under limited data. Our experiments on diverse LM downstream tasks show that LfU serves as an effective prior that enhances adaptability while preserving pretrained knowledge. For example, our LM from LfU achieves a 16.8% average improvement on math tasks compared to vanilla SFT on the same dataset, where the latter even leads to degraded performance on those tasks. Furthermore, LfU exhibits improved robustness to prompt variations, e.g., yielding a 92.1% lower standard deviation in output performances compared to SFT, highlighting its versatile effects.
BibTeX
@misc{nam2025learning,
  author = {Yunhun Nam and Jaehyung Kim and Jongheon Jeong},
  title = {{Learning from the Undesirable: Robust Adaptation of Language Models without Forgetting}},
  year = {2025},
  eprint = {2511.13052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.13052},
}
Figure