TL;DR
Low-Rank Adaptation (LoRA) is the go-to method for parameter-efficient fine-tuning, but its static updates apply uniformly to all inputs, leading to catastrophic forgetting. This paper introduces DISeL (Dynamic Input-Sensitive LoRA), which adds lightweight, input-dependent gates to individual rank-one components. By selectively activating updates only when needed, DISeL mitigates forgetting while maintaining high fine-tuning performance and offering unique interpretability insights.
What Problem Does This Paper Address?
Fine-tuning large language models (LLMs) on specialized tasks often comes at a steep cost: catastrophic forgetting, where the model loses its general capabilities acquired during pre-training. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA mitigate this compared to full fine-tuning, they don't eliminate it.
The authors identify a fundamental structural flaw: standard fine-tuning methods are input-agnostic. They apply the exact same weight correction ($\Delta$) to every single input, whether it is a specialized math problem (the fine-tuning domain) or a generic news article (the pre-training domain). This creates an inevitable compromise between learning the new task and retaining old knowledge. The paper addresses this by asking: Can we design an adapter that dynamically learns when to apply its corrections based on the input?
Methods
The authors propose DISeL (Dynamic Input-Sensitive LoRA). Instead of the standard LoRA update ($f(x) = ABx$), DISeL introduces a vector of input-dependent scalar gates $g(x) = \sigma(W_g x + b_g) \in [0, 1]^r$, scaling each rank-one component individually:
$$f(x) = AG(x)Bx = \sum_{i=1}^r g_i(x) a_i b_i^\top x$$
Crucially, the gate bias $b_g$ is initialized to a small negative value (e.g., $-3$), meaning the gates start nearly closed ($\sigma(-3) \approx 0.05$). This ensures that at the start of training, DISeL defaults to the pre-trained model's behavior. During training, the gates only open if doing so actively reduces the fine-tuning loss.
To optimize this behavior, the gate parameters ($W_g, b_g$) are treated as a distinct optimization problem and trained with a separate, substantially larger learning rate (typically $5\times$ larger) than the core LoRA matrices $A$ and $B$. This architecture mathematically approximates a Bayes-optimal predictor without requiring access to the original pre-training data during fine-tuning.
Datasets
The authors evaluated DISeL across multiple scales and domains:
- RoBERTa-base: Fine-tuned on 5 GLUE benchmark tasks (MNLI, SST-2, QNLI, CoLA, MRPC). Forgetting was evaluated using Masked Language Modeling (MLM) perplexity on BookCorpusOpen, CC-News, and WikiText-2.
- Llama 2-7B: Fine-tuned on MetaMathQA and evaluated on GSM8K for mathematical reasoning.
- Mistral-7B: Fine-tuned on Magicoder-Evol-Instruct-110K and evaluated on HumanEval for code generation.
- LLM Retention Suite: A comprehensive 14-benchmark evaluation suite covering commonsense reasoning, science QA, and general knowledge (including HellaSwag, MMLU, ARC, BoolQ, and TriviaQA) to rigorously measure catastrophic forgetting.
Key Insights and Ideas
- Decoupling Rank from Forgetting: In standard LoRA, increasing the rank ($r$) typically worsens forgetting because it adds more unconstrained capacity that alters the model's base behavior. DISeL completely breaks this trend; because gates keep unnecessary components closed, you can scale up the rank for better performance without sacrificing retention.
- The Gating "Aha" Moment: Analysis of the trained gates reveals they act as a continuous routing mechanism. For example, on a Llama model fine-tuned for math, the gates open wide for math inputs, stay mostly shut for general text, and open halfway for coding tasks—which naturally share logical and structured properties with mathematics.
- Localization of Knowledge: DISeL provides a built-in diagnostic tool to see where adaptation happens. The authors found that gate activations are highly non-uniform: adaptation concentrates heavily in the MLP up-projection layers (
up_proj) within the middle and late layers of the network, while attention value projections (v_proj) remain mostly closed.
Key Results
- RoBERTa on GLUE: Full fine-tuning caused MLM perplexity to skyrocket to $\sim150$, and standard LoRA/DoRA degraded it to between $30$ and $100$. DISeL maintained a perplexity of around $9$ (remarkably close to the pre-trained baseline of $6.4$) while matching or exceeding full fine-tuning accuracy.
- LLM Instruction Tuning: On Llama 2 (Math), DISeL achieved $\sim60\%$ accuracy on GSM8K while keeping retention accuracy within $0.5\%$ of the base model. On Mistral (Code), it exceeded $70\%$ fine-tuning accuracy while matching the base model's retention almost perfectly, outperforming standard LoRA, DoRA, and AdaLoRA at higher ranks.
Limitations and Future Work
- No Absolute Guarantees: While DISeL significantly reduces forgetting empirically, it does not mathematically guarantee the absolute preservation of all pre-trained capabilities.
- Weight Merging Challenge: A major practical benefit of standard LoRA is that the low-rank matrices can be permanently merged into the base weights ($W_0 + AB$) for zero-overhead inference. Because DISeL's gates are input-dependent, the adapter cannot be trivially folded back into the base weights, maintaining a small computational overhead during inference.
- Scope: The current work focuses entirely on single-task fine-tuning; extending this input-dependent gating mechanism to multi-task learning or sequential continual learning remains an open question.
Summary updated