Abstract
Training Multimodal Large Language Models (MLLMs) often involves a "tug-of-war" between the language backbone and the modality encoder, leading to unstable or biased learning. This paper introduces CoMMIT (Coordinated Multimodal Instruction Tuning), a framework featuring a dynamic learning rate scheduler and gradient regularization. By balancing the learning progress of both modules, CoMMIT achieves faster convergence and significantly improved performance across vision and audio tasks.
The Challenge: Imbalanced Multimodal Learning
Fine-tuning MLLMs like LLaVA or BLIP-2 requires coordinating two distinct components: a pre-trained Large Language Model (LLM) and a modality-specific feature encoder. Ideally, both should adapt synchronously. However, standard training often falls into two traps:
- Optimization Oscillation: If the learning rates are poorly matched, the optimization trajectory "zig-zags," leading to extreme learning inefficiency.
- Biased Learning: One module may dominate early. If the LLM adapts too quickly, the encoder's gradients may diminish, causing the model to rely on language priors rather than input data—a primary source of hallucinations.
The CoMMIT Framework
CoMMIT orchestrates the training process through three core mechanisms:
- Multimodal Balance Coefficient ($\kappa_t$): This metric acts as a "thermometer," measuring the relative impact of LLM updates versus encoder updates on the output distribution. A value of $\kappa_t \approx 1$ indicates balanced learning.
- Dynamic Learning Rate Scheduler: CoMMIT uses a moving average of $\kappa_t$ to adjust learning rates in real-time. If the encoder lags, its learning rate is boosted while the LLM's is dampened, maintaining equilibrium.
- Gradient Regularization: An auxiliary loss prevents "gradient diminishing" by encouraging sufficient distribution shifts, ensuring the balance coefficient remains estimable and the model continues to learn effectively.
Key Insights: Achieving Stability
A critical finding is that the common practice of using a synchronized (identical) learning rate for both modules often results in the highest oscillation. Conversely, simply increasing the learning rate for one module might smooth the curve but often leads to the complete neglect of the other component, resulting in inferior final performance.
Experimental Results
CoMMIT was evaluated across diverse architectures (BLIP-2, InternVL2, LLaVA-1.5, SALMONN) and modalities (Vision and Audio).
- Vision Performance: On the A-OKVQA task with BLIP-2, CoMMIT improved accuracy from 54.06% (baseline) to 64.37%.
- Audio Performance: For the SALMONN model on ClothoAQA, CoMMIT reached 50.55% accuracy, significantly outperforming the 42.49% baseline.
- Efficiency: The method demonstrates faster convergence, particularly on out-of-domain tasks like IconQA, where substantial adaptation is required.
Conclusion
CoMMIT provides an architecture-agnostic solution to the fundamental challenge of multimodal coordination. By dynamically balancing the learning process, it mitigates biases and oscillations, leading to more robust and capable MLLMs. Future work will explore its application to closed-source models and its interaction with various adapter mechanisms.
Summary updated