On Demand, Not Uniformity: Adaptive Expert Orchestration for Multimodal Continual Instruction Tuning

Mengxin Qin, Kun Wei, Weitao An, Jiahua Li, Xu Yang, Cheng DengarXiv 2026 · arXiv

TL;DR

CIRA-MoE is a novel framework for Multimodal Continual Instruction Tuning that replaces uniform model expansion with a "demand-aligned" approach. By identifying exactly where existing knowledge falls short at both the layer and token levels, it selectively injects new experts and arbitrates their contributions, significantly reducing catastrophic forgetting while improving adaptation to new tasks.

What Problem Does This Paper Address?

Multimodal Large Language Models (MLLMs) are powerful, but they struggle with Continual Learning: the ability to learn new tasks (like scientific QA or image captioning) sequentially without forgetting what they previously learned.

A popular solution is the Mixture-of-Experts (MoE) architecture, where the model is expanded by adding new "expert" modules for new tasks while freezing old ones. However, current MoE methods suffer from a "uniformity" problem. They typically add the same number of experts to every layer and treat every token in a sentence as equally difficult.

The authors of this paper discovered that adaptation demand is actually highly heterogeneous. As shown in Figure 1, some layers face much higher "gradient pressure" than others for specific tasks, and some tokens are easily handled by old experts while others are completely "uncovered." Adding capacity uniformly is inefficient and can lead to interference between old and new knowledge.

Figure 1. Layer- and token-wise heterogeneity in MCIT. (a) Layer- wise demand measured by gradient pressure. (b) Token-wise de- mand characterized by the distribution of teacher-forced NLL.

Methods

The authors propose the Coverage-Calibrated Expert Injection and Residual Arbitration (CIRA-MoE) framework. It consists of four key components:

  1. Expert-Wise Acceptance Routing (EWAR): Instead of standard top-$k$ routing, each expert has an independently learnable acceptance threshold. This produces a "signed acceptance margin"—a signal that tells the model how well an expert "covers" a specific input token.
  2. Layer-Wise Expert Injection (LWEI): Before training on a new task, the model probes a tiny subset (1%) of the data to see where historical experts fail. It calculates a "coverage gap" for each layer and only injects new experts into the layers that truly need more capacity.
  3. Token-Wise Residual Arbitration (TWRA): During training and inference, the model compares the "evidence" from old experts versus new ones. If old experts are already confident about a token, the new expert's contribution is dialed down. If the old experts are lost, the new expert takes the lead.
  4. Arbitration-Consistent Regularization (ACR): This includes two auxiliary losses that encourage experts to specialize and prevent new experts from intervening when they aren't needed.
Figure 2. Overview of CIRA-MoE. For each incoming multimodal instruction task, CIRA-MoE: (1) receives the latest task; (2) probes frozen historical experts using a 1% subset to estimate historical coverage cm(x) and layer-wise coverage gaps Sm; (3) injects up to two current experts per FFN projection according to Sm, allocating more capacity to poorly covered positions; and (4) applies Expert-Wise Acceptance Routing and Token-Wise Residual Arbitration to select experts and regulate current residual contributions. After training, current experts are frozen and retained with their stage indices; the same residual aggregation is used at inference.

Datasets

The framework was evaluated on two major benchmarks for multimodal continual learning:

  • UCIT: A sequence of 6 tasks including ImageNet-R (classification), ArxivQA (scientific diagrams), Vizcap (captioning), IconQA, CLEVR-Math, and Flickr30k.
  • CoIN: A more extensive sequence of 8 tasks including ScienceQA, VizWiz, REC, TextVQA, GQA, VQAv2, and OCR-VQA.

Key Insights and Ideas

  • Acceptance Margins as a Proxy for Demand: One of the most important "aha moments" was proving that the model's internal "acceptance margin" is a reliable proxy for actual learning difficulty. The authors showed a strong correlation between these margins and ground-truth metrics like gradient pressure and prediction error (NLL).
  • More is Not Always Better: The paper demonstrates that "starving" well-covered layers while over-parameterizing poorly-covered ones (the "Reversed-gap" strategy) actually hurts performance. Targeted injection is the key to efficiency.
  • Direction vs. Confidence: The "Arbitration" mechanism works by looking at both the direction (is the new expert better than the old?) and confidence (how much better?). Both are necessary to prevent the model from overwriting stable old knowledge with noisy new updates.
Figure 3. Historical coverage diagnostics. (a) Layer-wise coverage gap Sm versus normalized gradient pressure. (b) Signed historical acceptance margin cm(x) versus answer-token NLL.

Key Results

  • State-of-the-Art Performance: CIRA-MoE outperformed all existing baselines. On the UCIT benchmark, it achieved an average accuracy of 71.68%, beating the previous best method (HiDe-LLaVA) by 2.74 percentage points.
  • Superior Retention: It showed marked gains in "reasoning-heavy" tasks while maintaining the highest final retention on the very first tasks in the sequence, proving it effectively mitigates catastrophic forgetting.
  • Efficiency: Despite its adaptive nature, CIRA-MoE is computationally efficient. It adds fewer experts over time as the "pool" of knowledge grows and maintains a training throughput (TFLOPs/s) comparable to or better than standard MoE methods.
  • Task-Order Robustness: The model's performance remained stable regardless of the order in which tasks were presented, a critical requirement for real-world deployment.

Limitations and Future Work

While CIRA-MoE significantly slows down parameter growth, the expert pool still expands with every new task. The authors note that future work could explore more aggressive expert pruning or merging to keep the model size manageable over extremely long task sequences. Additionally, while the 1% probing step is efficient, exploring "on-the-fly" injection without a separate probing phase could further streamline the training process.

Summary updated

Abstract

Multimodal Continual Instruction Tuning (MCIT) enables Multimodal Large Language Models (MLLMs) to adapt to sequential instruction data while retaining previously acquired knowledge. While recent Mixture-of-Experts (MoE)-based methods mitigate forgetting by freezing historical experts and introducing new ones, they largely overlook the varying adaptation demands across tasks. Probing incoming tasks reveals pronounced heterogeneity in layer-wise gradient pressure and token-wise prediction difficulty. This observation highlights two unresolved decisions: where additional capacity is needed and how strongly new experts should contribute to individual tokens. To address both questions, we propose the Coverage-Calibrated Expert Injection and Residual Arbitration framework (CIRA-MoE), which derives expert acceptance margins via learnable threshold routing to serve as a unified coverage signal. Specifically, historical acceptance evidence guides the selective injection of new experts into insufficiently covered layers. Meanwhile, relative evidence between historical and current experts modulates token-wise residual contributions, coupled with arbitration-consistent regularization to promote expert specialization. Comprehensive experiments on UCIT and CoIN demonstrate that CIRA-MoE outperforms existing state-of-the-art methods in MCIT.
BibTeX
@misc{qin2026demand,
  author = {Mengxin Qin and Kun Wei and Weitao An and Jiahua Li and Xu Yang and Cheng Deng},
  title = {{On Demand, Not Uniformity: Adaptive Expert Orchestration for Multimodal Continual Instruction Tuning}},
  year = {2026},
  eprint = {2611.23865},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23865},
}
Figure