Introduction
As Multimodal Large Language Models (MLLMs) evolve, the ability to learn new tasks sequentially without degrading performance on previous ones—known as continual learning—becomes critical. However, MLLMs frequently suffer from catastrophic forgetting. The CoIN (Continual Instruction tuNing) benchmark is introduced to diagnose this phenomenon, revealing that forgetting in MLLMs is often not a loss of core reasoning ability, but rather a failure in instruction alignment.
The CoIN Benchmark and Dual-Metric Evaluation
The authors propose a comprehensive framework consisting of 10 datasets across 8 tasks, including General VQA, ScienceQA, OCR-VQA, and Grounding. To better understand the nature of forgetting, CoIN employs a Dual-Metric Evaluation strategy.
Traditional metrics often rely on exact string matches (Truth Alignment), which can be misleading if a model provides a correct answer in the wrong format. To address this, CoIN introduces Reasoning Capability evaluation, using a powerful LLM (e.g., Qwen-1.5-32B) as a judge to determine if the model's logic remains sound, even if the output format deviates from the instructions.
MoELoRA: Mitigating Forgetting
To combat catastrophic forgetting, the researchers introduced MoELoRA, an architecture combining Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA). By utilizing task-specific "experts" within the LoRA layers, the model can isolate new knowledge, preventing it from overwriting parameters essential for previously learned tasks.
Key Insights: Forgetting as an Alignment Failure
A pivotal finding of the CoIN benchmark is that MLLMs often retain the "knowledge" required for a task but lose the "discipline" to follow specific instruction formats. For example, after learning a new task, a model might correctly identify an object but fail to provide the single-word response requested, instead outputting a full sentence.
The evaluation shows that Reasoning Capability scores remain significantly more stable than Truth Alignment scores during sequential learning. This suggests that the model's cognitive foundations are robust, while its ability to align with specific output constraints is fragile.
Comparative Results
The benchmark reveals significant differences in how various MLLMs handle sequential tuning. While models like LLaVA and Qwen-VL exhibit substantial performance drops (with LLaVA showing a Backward Transfer of -32.62), MiniGPT-v2 demonstrates notable resilience.
Experimental results further validate that increasing the number of experts in MoELoRA significantly improves Mean Average Accuracy (MAA). For instance, LLaVA's MAA improved from 32.97 (with 1 expert) to 42.76 (with 8 experts), effectively reducing the impact of forgetting.
Conclusion
The CoIN benchmark provides a necessary diagnostic tool for the development of lifelong learning MLLMs. By distinguishing between reasoning and alignment, it shifts the focus of mitigation strategies toward maintaining instruction-following discipline. The success of MoELoRA highlights the potential of modular architectures in building more robust and adaptable multimodal systems.
Summary updated