CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models

Junda Wu, Xintong Li, Tong Yu, Yu Wang, Xiang Chen, Jiuxiang Gu, Lina Yao, Jingbo Shang, Julian J. McAuley

Abstract

Training Multimodal Large Language Models (MLLMs) often involves a "tug-of-war" between the language backbone and the modality encoder, leading to unstable or biased learning. This paper introduces CoMMIT (Coordinated Multimodal Instruction Tuning), a framework featuring a dynamic learning rate scheduler and gradient regularization. By balancing the learning progress of both modules, CoMMIT achieves faster convergence and significantly improved performance across vision and audio tasks.

The Challenge: Imbalanced Multimodal Learning

Fine-tuning MLLMs like LLaVA or BLIP-2 requires coordinating two distinct components: a pre-trained Large Language Model (LLM) and a modality-specific feature encoder. Ideally, both should adapt synchronously. However, standard training often falls into two traps:

  1. Optimization Oscillation: If the learning rates are poorly matched, the optimization trajectory "zig-zags," leading to extreme learning inefficiency.
  2. Biased Learning: One module may dominate early. If the LLM adapts too quickly, the encoder's gradients may diminish, causing the model to rely on language priors rather than input data—a primary source of hallucinations.
Conceptual illustration of learning inefficiencies: (a) oscillation due to imbalanced gradients and (b) insufficient learning when one module dominates.

The CoMMIT Framework

CoMMIT orchestrates the training process through three core mechanisms:

  • Multimodal Balance Coefficient ($\kappa_t$): This metric acts as a "thermometer," measuring the relative impact of LLM updates versus encoder updates on the output distribution. A value of $\kappa_t \approx 1$ indicates balanced learning.
  • Dynamic Learning Rate Scheduler: CoMMIT uses a moving average of $\kappa_t$ to adjust learning rates in real-time. If the encoder lags, its learning rate is boosted while the LLM's is dampened, maintaining equilibrium.
  • Gradient Regularization: An auxiliary loss prevents "gradient diminishing" by encouraging sufficient distribution shifts, ensuring the balance coefficient remains estimable and the model continues to learn effectively.

Key Insights: Achieving Stability

A critical finding is that the common practice of using a synchronized (identical) learning rate for both modules often results in the highest oscillation. Conversely, simply increasing the learning rate for one module might smooth the curve but often leads to the complete neglect of the other component, resulting in inferior final performance.

Comparison of the Multimodal Balance Coefficient ($\kappa_t$) across different strategies. CoMMIT maintains $\kappa_t$ closest to 1.0 with the lowest standard deviation.

Experimental Results

CoMMIT was evaluated across diverse architectures (BLIP-2, InternVL2, LLaVA-1.5, SALMONN) and modalities (Vision and Audio).

  • Vision Performance: On the A-OKVQA task with BLIP-2, CoMMIT improved accuracy from 54.06% (baseline) to 64.37%.
  • Audio Performance: For the SALMONN model on ClothoAQA, CoMMIT reached 50.55% accuracy, significantly outperforming the 42.49% baseline.
  • Efficiency: The method demonstrates faster convergence, particularly on out-of-domain tasks like IconQA, where substantial adaptation is required.

Conclusion

CoMMIT provides an architecture-agnostic solution to the fundamental challenge of multimodal coordination. By dynamically balancing the learning process, it mitigates biases and oscillations, leading to more robust and capable MLLMs. Future work will explore its application to closed-source models and its interaction with various adapter mechanisms.

Summary updated

BibTeX
@misc{wu2024commit,
  author = {Junda Wu and Xintong Li and Tong Yu and Yu Wang and Xiang Chen and Jiuxiang Gu and Lina Yao and Jingbo Shang and Julian J. McAuley},
  title = {{CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models}},
  year = {2024},
  eprint = {2407.20454},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20454},
}
Figure