CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model

Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Lianli Gao, Jingkuan SongarXiv 2024 · DOI · arXiv

Introduction

As Multimodal Large Language Models (MLLMs) evolve, the ability to learn new tasks sequentially without degrading performance on previous ones—known as continual learning—becomes critical. However, MLLMs frequently suffer from catastrophic forgetting. The CoIN (Continual Instruction tuNing) benchmark is introduced to diagnose this phenomenon, revealing that forgetting in MLLMs is often not a loss of core reasoning ability, but rather a failure in instruction alignment.

The CoIN Benchmark and Dual-Metric Evaluation

The authors propose a comprehensive framework consisting of 10 datasets across 8 tasks, including General VQA, ScienceQA, OCR-VQA, and Grounding. To better understand the nature of forgetting, CoIN employs a Dual-Metric Evaluation strategy.

Traditional metrics often rely on exact string matches (Truth Alignment), which can be misleading if a model provides a correct answer in the wrong format. To address this, CoIN introduces Reasoning Capability evaluation, using a powerful LLM (e.g., Qwen-1.5-32B) as a judge to determine if the model's logic remains sound, even if the output format deviates from the instructions.

Figure 6: The CoIN framework illustrates continual training across tasks and the dual-metric evaluation process, distinguishing between surface-level Truth Alignment and underlying Reasoning Capability.

MoELoRA: Mitigating Forgetting

To combat catastrophic forgetting, the researchers introduced MoELoRA, an architecture combining Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA). By utilizing task-specific "experts" within the LoRA layers, the model can isolate new knowledge, preventing it from overwriting parameters essential for previously learned tasks.

Key Insights: Forgetting as an Alignment Failure

A pivotal finding of the CoIN benchmark is that MLLMs often retain the "knowledge" required for a task but lose the "discipline" to follow specific instruction formats. For example, after learning a new task, a model might correctly identify an object but fail to provide the single-word response requested, instead outputting a full sentence.

The evaluation shows that Reasoning Capability scores remain significantly more stable than Truth Alignment scores during sequential learning. This suggests that the model's cognitive foundations are robust, while its ability to align with specific output constraints is fragile.

Comparative Results

The benchmark reveals significant differences in how various MLLMs handle sequential tuning. While models like LLaVA and Qwen-VL exhibit substantial performance drops (with LLaVA showing a Backward Transfer of -32.62), MiniGPT-v2 demonstrates notable resilience.

Figure 4: Radar charts comparing the performance of LLaVA, Qwen-VL, and MiniGPT-v2 across the eight tasks in the CoIN benchmark.

Experimental results further validate that increasing the number of experts in MoELoRA significantly improves Mean Average Accuracy (MAA). For instance, LLaVA's MAA improved from 32.97 (with 1 expert) to 42.76 (with 8 experts), effectively reducing the impact of forgetting.

Conclusion

The CoIN benchmark provides a necessary diagnostic tool for the development of lifelong learning MLLMs. By distinguishing between reasoning and alignment, it shifts the focus of mitigation strategies toward maintaining instruction-following discipline. The success of MoELoRA highlights the potential of modular architectures in building more robust and adaptable multimodal systems.

Summary updated

Abstract

Instruction tuning represents a prevalent strategy employed by Multimodal Large Language Models (MLLMs) to align with human instructions and adapt to new tasks. Nevertheless, MLLMs encounter the challenge of adapting to users' evolving knowledge and demands. Therefore, how to retain existing skills while acquiring new knowledge needs to be investigated. In this paper, we present a comprehensive benchmark, namely Continual Instruction tuNing (CoIN), to assess existing MLLMs in the sequential instruction tuning paradigm. CoIN comprises 10 commonly used datasets spanning 8 task categories, ensuring a diverse range of instructions and tasks. Besides, the trained model is evaluated from two aspects: Instruction Following and General Knowledge, which assess the alignment with human intention and knowledge preserved for reasoning, respectively. Experiments on CoIN demonstrate that current powerful MLLMs still suffer catastrophic forgetting, and the failure in intention alignment assumes the main responsibility, instead of the knowledge forgetting. To this end, we introduce MoELoRA to MLLMs which is effective to retain the previous instruction alignment. Experimental results consistently illustrate the forgetting decreased from this method on CoIN.
BibTeX
@misc{chen2024coin,
  author = {Cheng Chen and Junchen Zhu and Xu Luo and Hengtao Shen and Lianli Gao and Jingkuan Song},
  title = {{CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model}},
  year = {2024},
  eprint = {2403.08350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.08350},
}
Figure