Learning When to Adapt

Ali Zindari, Xiaowen Jiang, Rotem Mulayoff, Sebastian U. SticharXiv 2026 · arXiv

TL;DR

Low-Rank Adaptation (LoRA) is the go-to method for parameter-efficient fine-tuning, but its static updates apply uniformly to all inputs, leading to catastrophic forgetting. This paper introduces DISeL (Dynamic Input-Sensitive LoRA), which adds lightweight, input-dependent gates to individual rank-one components. By selectively activating updates only when needed, DISeL mitigates forgetting while maintaining high fine-tuning performance and offering unique interpretability insights.

What Problem Does This Paper Address?

Fine-tuning large language models (LLMs) on specialized tasks often comes at a steep cost: catastrophic forgetting, where the model loses its general capabilities acquired during pre-training. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA mitigate this compared to full fine-tuning, they don't eliminate it.

The authors identify a fundamental structural flaw: standard fine-tuning methods are input-agnostic. They apply the exact same weight correction ($\Delta$) to every single input, whether it is a specialized math problem (the fine-tuning domain) or a generic news article (the pre-training domain). This creates an inevitable compromise between learning the new task and retaining old knowledge. The paper addresses this by asking: Can we design an adapter that dynamically learns when to apply its corrections based on the input?

Methods

The authors propose DISeL (Dynamic Input-Sensitive LoRA). Instead of the standard LoRA update ($f(x) = ABx$), DISeL introduces a vector of input-dependent scalar gates $g(x) = \sigma(W_g x + b_g) \in [0, 1]^r$, scaling each rank-one component individually:

$$f(x) = AG(x)Bx = \sum_{i=1}^r g_i(x) a_i b_i^\top x$$

Crucially, the gate bias $b_g$ is initialized to a small negative value (e.g., $-3$), meaning the gates start nearly closed ($\sigma(-3) \approx 0.05$). This ensures that at the start of training, DISeL defaults to the pre-trained model's behavior. During training, the gates only open if doing so actively reduces the fine-tuning loss.

To optimize this behavior, the gate parameters ($W_g, b_g$) are treated as a distinct optimization problem and trained with a separate, substantially larger learning rate (typically $5\times$ larger) than the core LoRA matrices $A$ and $B$. This architecture mathematically approximates a Bayes-optimal predictor without requiring access to the original pre-training data during fine-tuning.

Datasets

The authors evaluated DISeL across multiple scales and domains:

  • RoBERTa-base: Fine-tuned on 5 GLUE benchmark tasks (MNLI, SST-2, QNLI, CoLA, MRPC). Forgetting was evaluated using Masked Language Modeling (MLM) perplexity on BookCorpusOpen, CC-News, and WikiText-2.
  • Llama 2-7B: Fine-tuned on MetaMathQA and evaluated on GSM8K for mathematical reasoning.
  • Mistral-7B: Fine-tuned on Magicoder-Evol-Instruct-110K and evaluated on HumanEval for code generation.
  • LLM Retention Suite: A comprehensive 14-benchmark evaluation suite covering commonsense reasoning, science QA, and general knowledge (including HellaSwag, MMLU, ARC, BoolQ, and TriviaQA) to rigorously measure catastrophic forgetting.

Key Insights and Ideas

  • Decoupling Rank from Forgetting: In standard LoRA, increasing the rank ($r$) typically worsens forgetting because it adds more unconstrained capacity that alters the model's base behavior. DISeL completely breaks this trend; because gates keep unnecessary components closed, you can scale up the rank for better performance without sacrificing retention.
  • The Gating "Aha" Moment: Analysis of the trained gates reveals they act as a continuous routing mechanism. For example, on a Llama model fine-tuned for math, the gates open wide for math inputs, stay mostly shut for general text, and open halfway for coding tasks—which naturally share logical and structured properties with mathematics.
Gate activation histograms across domains showing selective opening in middle and late layers.
  • Localization of Knowledge: DISeL provides a built-in diagnostic tool to see where adaptation happens. The authors found that gate activations are highly non-uniform: adaptation concentrates heavily in the MLP up-projection layers (up_proj) within the middle and late layers of the network, while attention value projections (v_proj) remain mostly closed.

Key Results

  • RoBERTa on GLUE: Full fine-tuning caused MLM perplexity to skyrocket to $\sim150$, and standard LoRA/DoRA degraded it to between $30$ and $100$. DISeL maintained a perplexity of around $9$ (remarkably close to the pre-trained baseline of $6.4$) while matching or exceeding full fine-tuning accuracy.
Figure 2: ROBERTa on GLUE. We fine-tune ROBERTa-base on five GLUE tasks: MNLI, SST-2, QNLI, COLA, and MRPC, using various methods and ranks. The vertical axis measures the average test accuracy for the corresponding tasks, while the horizontal axis indicates the average masked-LM perplexity, evaluated on BOOKCORPUSOPEN, CC-NEWS, and WIKITEXT-2. Each point represents a method-rank pair averaged across three seeds, with error bars indicating the standard deviation. The dashed line marks the pre-trained baseline. Here, DISeL and ADALORA remain close to this baseline while matching the FT accuracy of full FT, whereas the other methods forget much more.
  • LLM Instruction Tuning: On Llama 2 (Math), DISeL achieved $\sim60\%$ accuracy on GSM8K while keeping retention accuracy within $0.5\%$ of the base model. On Mistral (Code), it exceeded $70\%$ fine-tuning accuracy while matching the base model's retention almost perfectly, outperforming standard LoRA, DoRA, and AdaLoRA at higher ranks.

Limitations and Future Work

  • No Absolute Guarantees: While DISeL significantly reduces forgetting empirically, it does not mathematically guarantee the absolute preservation of all pre-trained capabilities.
  • Weight Merging Challenge: A major practical benefit of standard LoRA is that the low-rank matrices can be permanently merged into the base weights ($W_0 + AB$) for zero-overhead inference. Because DISeL's gates are input-dependent, the adapter cannot be trivially folded back into the base weights, maintaining a small computational overhead during inference.
  • Scope: The current work focuses entirely on single-task fine-tuning; extending this input-dependent gating mechanism to multi-task learning or sequential continual learning remains an open question.

Summary updated

Abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This input-agnostic approach creates an inevitable compromise between adapting to the fine-tuning distribution and preserving pre-trained behavior on inputs outside that distribution, contributing to catastrophic forgetting. We introduce DISeL (Dynamic Input-Sensitive LoRA), which augments LoRA modules with lightweight input-dependent gates over individual rank-one components. The gating mechanism is designed to preserve the pre-trained model's behavior by default, while training learns to activate selected components that reduce the fine-tuning loss. DISeL adds only a small number of parameters and preserves the low-rank structure. Across RoBERTa on GLUE, and Llama and Mistral models fine-tuned for mathematical reasoning and code generation, DISeL reduces forgetting relative to LoRA and related variants while maintaining competitive fine-tuning accuracy. In addition, the learned gate activations provide an interpretable diagnostic view of which layers and rank components are most activated during fine-tuning, giving insight into where task-specific adaptation is concentrated. Code available at https://github.com/alizindari/DISeL .
BibTeX
@misc{zindari2026learning,
  author = {Ali Zindari and Xiaowen Jiang and Rotem Mulayoff and Sebastian U. Stich},
  title = {{Learning When to Adapt}},
  year = {2026},
  eprint = {2605.19028},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.19028},
}

Loading related content...

Figure