Introduction
Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities in following instructions and reasoning about visual and textual data. However, when trained on new tasks sequentially, they suffer from a phenomenon known as "catastrophic forgetting." This paper introduces CoIN, the first comprehensive benchmark specifically designed for continual instruction tuning in MLLMs. Through CoIN, the authors reveal a surprising insight: models do not necessarily forget their underlying knowledge; rather, they forget how to format their answers. To mitigate this, the authors propose MoELoRA, a mixture-of-experts approach that helps models retain previous instruction alignment while acquiring new skills.
The Challenge of Continual Learning
In an ideal scenario, an AI model should be a lifelong learner. It should be able to learn a new task—such as reading text in images (OCR)—without forgetting how to perform a previously learned task, like identifying objects or answering science questions.
Unfortunately, most MLLMs are trained in static batches. When updated sequentially, the neural network's weights shift so drastically to accommodate the new task that the old capabilities are effectively overwritten. While catastrophic forgetting has been extensively studied in traditional computer vision and standard LLMs, there has been no dedicated benchmark or deep analysis for continual instruction tuning in multimodal models. The authors sought to answer a critical question: what exactly is being forgotten, and how can we prevent it?
Methodology: The CoIN Benchmark and Dual-Track Evaluation
The researchers tackled this problem through three primary contributions:
- The CoIN Benchmark: They curated a comprehensive suite of 10 high-quality datasets across 8 distinct tasks to simulate a sequential "curriculum" for an MLLM. These tasks include Grounding (RefCOCO variants), Classification (ImageNet), General VQA (VQAv2, GQA), Knowledge-Grounded IQA (ScienceQA), OCR/Reading (TextVQA, OCR-VQA), and Specialized Tasks (VizWiz).
- Dual-Track Evaluation: This represents the paper's most innovative methodological pivot. Instead of solely measuring traditional accuracy (which they term Truth Alignment), they utilized a powerful "teacher" LLM (Qwen-1.5-32B) to evaluate the model's underlying logic and internal knowledge (Reasoning Capability). This dual-track approach distinguishes between a model that fails because it forgot the facts versus a model that fails because it forgot the required output format (e.g., answering with a full sentence instead of a requested single word).
- MoELoRA: To combat forgetting, they introduced a novel architecture combining Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA). Instead of using one set of trainable weights for all tasks, MoELoRA employs multiple "experts" managed by a gating function. When a model encounters a specific task, the gating function activates only the most relevant experts, drastically reducing interference between old and new knowledge.
Key Insights: The "Interface" Problem
The most significant finding of the study is that MLLMs primarily lose their instruction alignment, not their reasoning knowledge.
When tested on older tasks, traditional accuracy metrics (Truth Alignment) showed massive performance drops. For example, LLaVA's performance on grounding tasks plummeted from 31% to 0%. However, the LLM-based "Reasoning Capability" score showed much smaller declines. This indicates that the model's "brain" (knowledge) is still intact, but its "mouth" (output formatting) is confused.
Furthermore, the authors found that instruction diversity acts as a shield. Using a wider variety of instruction templates during training makes the model more robust. If a model learns to answer a question in ten different ways, its formatting is less likely to be overwritten by a single new task. Additionally, the order in which tasks are learned significantly impacts final performance.
Results: Catastrophic Forgetting and the MoELoRA Solution
The evaluation confirmed that catastrophic forgetting is a severe issue for standard architectures. Popular models like LLaVA and Qwen-VL showed massive degradation on previously learned tasks. LLaVA exhibited a staggering Backward Transfer (BWT)—a metric measuring the loss of old performance—of -32.62.
Interestingly, MiniGPT-v2 showed an anomalous lack of forgetting (BWT of +6.04). However, the authors noted this was largely due to "underfitting"; the model struggled to learn the new tasks effectively in the first place, which inadvertently protected its older capabilities.
By implementing MoELoRA with 8 experts, the researchers successfully mitigated these issues. MoELoRA improved LLaVA’s Mean Average Accuracy (MAA) from 32.97 to 42.76 and significantly reduced forgetting. The study also found a direct correlation between the number of experts and performance: more experts allow for better specialization, preventing the model's parameters from tangling when switching between vastly different tasks like ScienceQA and OCR.
Limitations and Future Directions
While CoIN and MoELoRA represent significant steps forward, the authors note a few limitations:
- Model Scale: The study focused on models in the 7B to 9B parameter range. It remains an open question whether massive models (70B+) exhibit the same "format-only" forgetting.
- Architecture Bias: Most tested models were based on the LLaMA architecture, suggesting a need to explore different backbones.
- Task Breadth: Real-world multimodal interaction involves complexities like video and audio that CoIN's current 8 tasks do not yet cover.
Ultimately, the authors suggest that future research in continual learning should pivot toward "instruction-preserving" techniques rather than solely focusing on "knowledge-preserving" ones, given that instruction alignment is the first capability to degrade.
Summary updated