CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

Tao Hu, Ling-Xiang Zhang, Da-Wei ZhouarXiv 2026 · arXiv

TL;DR

CoPre-MoE is a novel framework for Multimodal Continual Instruction Tuning (MCIT) that solves the "parameter explosion" problem in Mixture-of-Experts (MoE) architectures. By selectively consolidating redundant cross-task capabilities into a fixed-capacity generic pool and compressing task-specific features into a private pool, it achieves state-of-the-art performance with 68% fewer parameters than existing methods while effectively eliminating catastrophic forgetting.

What Problem Does This Paper Address?

Multimodal Large Language Models (MLLMs) are increasingly expected to learn new tasks—like visual reasoning or document QA—sequentially. However, standard fine-tuning leads to catastrophic forgetting, where new knowledge overwrites the old.

While Mixture-of-Experts (MoE) architectures mitigate this by assigning specific "experts" to different tasks, they face a dual dilemma:

  1. Linear Bloat: Adding new experts for every task causes the model's parameter count to skyrocket.
  2. Functional Redundancy: Many tasks share underlying skills. For example, both ScienceQA and TextVQA require the model to locate specific image regions. Existing MoE methods ignore this, training redundant experts that perform nearly identical functions.

The authors demonstrate this redundancy empirically in Figure 1, showing that swapping a task's native expert with a similar one from a different task often preserves 90–100% of the performance.

Figure 1. Empirical analysis of cross-task expert redundancy on UCIT. Comparing native experts (Before Swap) with cross- task substitution (After Overlap Swap) shows that high-overlap experts preserve 90% ∼100% of performance across tasks, con- firming substantial functional redundancy while highlighting the need to preserve task-specific variations.

Methods

CoPre-MoE introduces a selective dual-pool architecture to balance efficiency and knowledge preservation. The process follows a "train-then-consolidate" workflow:

  1. Task-Specific Training: For a new task, the model trains low-rank experts and a router on a frozen MLLM backbone.
  2. Selective Generic Consolidation: At task boundaries, the model evaluates functional overlap using activation statistics. Experts with high overlap are merged into a fixed-capacity Generic Pool using "activation-weighted operator merging." This ensures that shared skills are stored once rather than duplicated.
  3. Compact Private Preservation: Experts with unique, task-specific capabilities are moved to a Private Pool. To keep this pool lean, CoPre-MoE uses a Tucker-style decomposition, factorizing experts into shared projection bases and compact task-specific cores.
  4. Orthogonal Key-Value Routing: To prevent the model from getting "confused" between tasks at inference time, the router uses learnable keys. New keys are regularized to be orthogonal to historical ones, allowing the model to dynamically query both pools without needing explicit task labels.
Figure 2. Overview of CoPre-MoE. Left: Task-specific low-rank experts and key-value routing pairs are trained on a frozen MLLM backbone. Middle: At task boundaries, functionally overlapping experts are consolidated into a fixed-capacity Generic Pool, while task- specific experts are factorized into shared bases and compact cores in a Private Pool. Right: Routing query q dynamically queries both pools to compose generic and private operators for inference.

Datasets

The authors evaluated CoPre-MoE on two major sequential benchmarks:

  • UCIT: A 6-task sequence including ImageNet-R, ArxivQA, VizWiz, IconQA, CLEVR, and Flickr30k.
  • CoIN: A longer 8-task sequence covering ScienceQA, TextVQA, ImageNet, GQA, VizWiz, Grounding (RefCOCO), VQAv2, and OCR-VQA.

Key Insights and Ideas

  • Redundancy is an Opportunity: The core "aha moment" is that cross-task redundancy isn't just a waste of space—it's a signal. By merging these redundant experts, the model naturally builds a "generic" knowledge base that aids future learning.
  • Decoupling Works: The t-SNE visualizations in Figure 4 (represented as art_7ab9e38ffc2e4263944bcf7025fea3f3 in the metadata) show that the Generic Pool forms a continuous manifold of shared features, while the Private Pool keeps task-specific features in well-separated clusters. This structural separation is key to preventing interference.
  • Activation Weighting Matters: Simply averaging expert weights during merging degrades performance. Using activation-weighted merging preserves the specific feature channels that are most critical to the model's output.

Key Results

CoPre-MoE sets a new benchmark for multimodal continual learning:

  • Superior Retention: It achieved a Backward Transfer (BWT) score of -0.10 on UCIT, indicating almost zero forgetting of previous tasks. As shown in Figure 3, while other methods like MoELoRA and SEFE degrade as the task sequence lengthens, CoPre-MoE remains remarkably stable.
  • Efficiency: Compared to MoELoRA, it reduces adaptation parameter overhead by over 68%.
  • Inference Speed: By routing through compact cores with shared projections, it reduces latency by 44.1% and boosts decoding throughput compared to the strongest baselines.
Figure 3. Backward transfer trajectory across sequential training stages t on UCIT. Points evaluate average BWT over previously learned tasks; terminal values report final retention.

Limitations and Future Work

The authors acknowledge that while CoPre-MoE preserves task capabilities, it still inherits the intrinsic limitations of its underlying MLLM backbone, such as visual hallucinations and safety alignment issues. Future work could explore how to integrate safety alignment into the continual learning process itself. Additionally, while the generic pool has a fixed capacity, extremely long task streams might eventually saturate it, suggesting a need for dynamic capacity management in future iterations.

Summary updated

Abstract

Multimodal continual instruction tuning requires multimodal large language models to sequentially acquire new capabilities without catastrophic forgetting. Mixture-of-experts (MoE) architectures offer a promising way to mitigate task interference through expert specialization and combination. However, while existing MoE-based methods avoid overwriting shared parameters by allocating separate experts for each task, this progressive expansion incurs severe cross-task redundancy and linear parameter growth. Specifically, experts across tasks often learn overlapping functions alongside task-specific capabilities. For instance, ScienceQA and TextVQA both locate relevant image regions. To curb parameter redundancy without inducing forgetting, we propose Consolidating and Preserving MoE (CoPre-MoE). For each incoming task, CoPre-MoE first learns task-specific MoE to capture specialized capabilities. To eliminate redundancy and reduce parameter growth across tasks, we design a shared-and-private expert architecture. Specifically, we first evaluate functional overlap to selectively consolidate overlapping experts across tasks into a fixed-capacity generic expert pool. We then factorize the remaining task-specific experts into shared projection bases and compact task-specific cores within a private expert pool. To mitigate forgetting, we construct the router by pairing each expert with learnable retrieval keys, where historical keys remain frozen during continual updates while newly added keys are regularized with an orthogonality penalty. At inference, input-dependent routing dynamically combines both pools without test-time task labels. Extensive experiments demonstrate that CoPre-MoE achieves superior performance while substantially reducing parameter overhead.
BibTeX
@misc{hu2026copre,
  author = {Tao Hu and Ling-Xiang Zhang and Da-Wei Zhou},
  title = {{CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning}},
  year = {2026},
  eprint = {2611.18417},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18417},
}
Figure