TL;DR
Fine-tuning Large Multimodal Models (LMMs) on narrow tasks often triggers catastrophic performance drops on general benchmarks, but this paper reveals that much of this "forgetting" is actually a temporary output-distribution bias rather than erased knowledge. By tracing how model layers alter output token preferences, the authors introduce two simple, parameter-efficient fine-tuning recipes—updating only the self-attention projections or only the MLP Gate & Up layers—that achieve strong skill acquisition with virtually zero held-out degradation across multiple model families.
What Problem Does This Paper Address?
Training LMMs from scratch costs millions of dollars and weeks of compute. However, sequentially fine-tuning an existing model on specialized skills (such as medical VQA, reading clocks, or counting objects) typically causes severe catastrophic forgetting across general visual and reasoning benchmarks.
Recent literature debated whether LMMs inherently suffer from irreversible forgetting. This paper investigates:
- Where in the model adaptation should occur to maximize learning while minimizing forgetting.
- Why forgetting occurs at a mechanistic level.
- Whether forgotten capabilities can be recovered without storing old training data (rehearsal).
Methods
The authors systematically evaluate sequential continual learning without data replay across three model backbones (LLaVA-OneVision 7B, LLaVA-NeXT 8B, and Qwen2.5-VL 7B).
Instead of treating the language decoder as a monolithic block, they dissect transformer sub-layers into routing and memory-writing mechanisms:
- Selective Component Tuning: Comparing full fine-tuning, vision-side tuning, and projector tuning against surgical interventions inside the LLM:
- SA Proj.: Tuning only self-attention projection weights ($W_Q, W_K, W_V, W_O$).
- MLP (Gate & Up): Tuning only the concept-activation layers ($W_{\text{gate}}, W_{\text{up}}$) while freezing the write-back down-projection matrix ($W_{\text{down}}$).
- Counting-Bias Output Probing: A diagnostic tool that measures the model's next-token probability for numeric tokens on general, non-counting captions (from LCS-558K).
- Layer-Wise Residual-to-Logit Contribution Analysis: Registering forward hooks across layers and projecting residual updates through the unembedding head to measure which layers and submodules drive the distribution drift.
Datasets
- Target Task Suite (Sequential Curriculum):
- CUB-200: Fine-grained bird species classification (reformulated as multiple-choice VQA).
- PixmoCount: Object counting.
- PathVQA: Radiology / medical question answering.
- TextVQA: Visual OCR reading.
- TimeClock: Analogue clock time reading.
- Held-Out General Benchmark Suite (8 benchmarks): AI2D, ChartQA, DocVQA, InfoVQA, RealWorldQA, SeedBench, ScienceQA, and MMStar.
- Diagnostic Data: LCS-558K image-captioning subsets used for tracking token distribution drift.
Key Insights and Ideas
- Forgetting is largely an output distribution shift: When fine-tuned on a narrow task like counting, the model develops an overwhelming bias toward generating numeric tokens—even describing a general photo as "There are 2 boots in this image" instead of generating a rich caption. The visual understanding remains intact; the model has simply acquired an output shortcut.
- "Forgotten" abilities can spontaneously recover: When the model is subsequently trained on a task with varied, non-numeric outputs (like PathVQA), the numeric bias recedes, and held-out benchmark scores rebound significantly without any rehearsal of previous data.
- Late MLP layers drive the drift: Layer-wise residual analysis demonstrates that self-attention updates act as routing mechanisms with flat, minor logit-space changes. In contrast, the final 4–5 MLP layers drive almost the entire shift in vocabulary output logits.
- Freezing $W_{\text{down}}$ stops the bias: Because $W_{\text{down}}$ writes representations directly back into the residual stream toward the vocabulary head, keeping it frozen while updating $W_{\text{gate}}$ and $W_{\text{up}}$ allows the model to learn new task concepts without letting them hijack the global output distribution.
Key Results
- Self-Attention Projections (
SA Proj.): - On LLaVA-OneVision, yields +24.9 percentage points (pp) target learning with only -0.6 pp held-out degradation across the 5-task sequence (vs. -27.4 pp for full fine-tuning and -23.3 pp for full LLM tuning).
- On Qwen2.5-VL, held-out performance actually improved (+0.6 pp) alongside a +14.9 pp net target gain.
- MLP Gate & Up (
MLP Gate & Up): - Delivers the strongest balance: +30.5 pp target learning and +27.1 pp target overall on LLaVA-OneVision, while restricting held-out forgetting to just -2.1 pp (compared to -15.7 pp when tuning the entire MLP).
- Comparison to Existing Baselines:
- Matches or outperforms established continual learning techniques like LoRA, Mixture-of-Experts (MoE), weight ensembling (WiSE-FT), and Learning without Forgetting (LwF), while requiring no replay memory, no auxiliary teacher models, and no custom inference routing.
- SA Proj. achieved the highest throughput (1.46 samples/sec) with only 0.82B trainable parameters.
Limitations and Future Work
- Model Scale: Evaluations were conducted on 7B–8B scale models; behavior on frontier 70B+ models remains to be verified.
- Curriculum Length: Experiments examined five sequential stages; longer, open-ended curricula could eventually accumulate drift.
- Incomplete Baseline Recovery: Subsequent tasks partially restore held-out scores, but models do not fully return to their original baseline due to subtle overlapping biases introduced by later tasks.
- Modalities: Explored purely within vision-language contexts; audio and video temporal adaptation were left for future research.