TL;DR
CIRA-MoE is a novel framework for Multimodal Continual Instruction Tuning that replaces uniform model expansion with a "demand-aligned" approach. By identifying exactly where existing knowledge falls short at both the layer and token levels, it selectively injects new experts and arbitrates their contributions, significantly reducing catastrophic forgetting while improving adaptation to new tasks.
What Problem Does This Paper Address?
Multimodal Large Language Models (MLLMs) are powerful, but they struggle with Continual Learning: the ability to learn new tasks (like scientific QA or image captioning) sequentially without forgetting what they previously learned.
A popular solution is the Mixture-of-Experts (MoE) architecture, where the model is expanded by adding new "expert" modules for new tasks while freezing old ones. However, current MoE methods suffer from a "uniformity" problem. They typically add the same number of experts to every layer and treat every token in a sentence as equally difficult.
The authors of this paper discovered that adaptation demand is actually highly heterogeneous. As shown in Figure 1, some layers face much higher "gradient pressure" than others for specific tasks, and some tokens are easily handled by old experts while others are completely "uncovered." Adding capacity uniformly is inefficient and can lead to interference between old and new knowledge.
Methods
The authors propose the Coverage-Calibrated Expert Injection and Residual Arbitration (CIRA-MoE) framework. It consists of four key components:
- Expert-Wise Acceptance Routing (EWAR): Instead of standard top-$k$ routing, each expert has an independently learnable acceptance threshold. This produces a "signed acceptance margin"—a signal that tells the model how well an expert "covers" a specific input token.
- Layer-Wise Expert Injection (LWEI): Before training on a new task, the model probes a tiny subset (1%) of the data to see where historical experts fail. It calculates a "coverage gap" for each layer and only injects new experts into the layers that truly need more capacity.
- Token-Wise Residual Arbitration (TWRA): During training and inference, the model compares the "evidence" from old experts versus new ones. If old experts are already confident about a token, the new expert's contribution is dialed down. If the old experts are lost, the new expert takes the lead.
- Arbitration-Consistent Regularization (ACR): This includes two auxiliary losses that encourage experts to specialize and prevent new experts from intervening when they aren't needed.
Datasets
The framework was evaluated on two major benchmarks for multimodal continual learning:
- UCIT: A sequence of 6 tasks including ImageNet-R (classification), ArxivQA (scientific diagrams), Vizcap (captioning), IconQA, CLEVR-Math, and Flickr30k.
- CoIN: A more extensive sequence of 8 tasks including ScienceQA, VizWiz, REC, TextVQA, GQA, VQAv2, and OCR-VQA.
Key Insights and Ideas
- Acceptance Margins as a Proxy for Demand: One of the most important "aha moments" was proving that the model's internal "acceptance margin" is a reliable proxy for actual learning difficulty. The authors showed a strong correlation between these margins and ground-truth metrics like gradient pressure and prediction error (NLL).
- More is Not Always Better: The paper demonstrates that "starving" well-covered layers while over-parameterizing poorly-covered ones (the "Reversed-gap" strategy) actually hurts performance. Targeted injection is the key to efficiency.
- Direction vs. Confidence: The "Arbitration" mechanism works by looking at both the direction (is the new expert better than the old?) and confidence (how much better?). Both are necessary to prevent the model from overwriting stable old knowledge with noisy new updates.
Key Results
- State-of-the-Art Performance: CIRA-MoE outperformed all existing baselines. On the UCIT benchmark, it achieved an average accuracy of 71.68%, beating the previous best method (HiDe-LLaVA) by 2.74 percentage points.
- Superior Retention: It showed marked gains in "reasoning-heavy" tasks while maintaining the highest final retention on the very first tasks in the sequence, proving it effectively mitigates catastrophic forgetting.
- Efficiency: Despite its adaptive nature, CIRA-MoE is computationally efficient. It adds fewer experts over time as the "pool" of knowledge grows and maintains a training throughput (TFLOPs/s) comparable to or better than standard MoE methods.
- Task-Order Robustness: The model's performance remained stable regardless of the order in which tasks were presented, a critical requirement for real-world deployment.
Limitations and Future Work
While CIRA-MoE significantly slows down parameter growth, the expert pool still expands with every new task. The authors note that future work could explore more aggressive expert pruning or merging to keep the model size manageable over extremely long task sequences. Additionally, while the 1% probing step is efficient, exploring "on-the-fly" injection without a separate probing phase could further streamline the training process.
Summary updated