Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models

Xinwei Li, Li Lin, Shuai Wang, Chen QianarXiv 2023 · DOI · arXiv

Abstract

Multi-modal Large Language Models (MLLMs) typically rely on instruction tuning, where a smaller student model learns from a larger teacher. However, traditional distillation is a unidirectional process that often ignores the student's specific learning needs. This paper introduces Competitive Multi-modal Distillation (CoMD), a framework that incorporates a bidirectional feedback loop. By identifying tasks the student finds difficult, the teacher can generate a personalized, challenging curriculum. Experimental results show that a 7B-parameter student trained via CoMD can outperform its 13B-parameter teacher on benchmarks like ScienceQA and the LLaVA test set.

Introduction

The prevailing method for developing efficient Multi-Modal Large Language Models (MLLMs) involves instruction tuning—distilling knowledge from a large teacher model into a more compact student model. Despite its success, this approach suffers from two primary limitations. First, it is a "one-way street" where the teacher provides data without regard for the student's current proficiency or specific struggles. Second, generating high-quality multi-modal instruction data is resource-intensive, often requiring significant manual effort or expensive API calls to proprietary models.

Researchers propose that the distillation process should be treated as a personalized tutoring session rather than a static lecture. By enabling mutual feedback, the teacher can recognize when a student struggles with specific concepts—such as identifying a particular character in a complex scene—and provide targeted examples to address those weaknesses.

The CoMD Framework

The Competitive Multi-modal Distillation (CoMD) framework operates through a two-stage process designed to optimize the student's learning trajectory.

Stage 1: Multi-modal Pre-training

Initially, the student model undergoes feature alignment. Using a filtered dataset of approximately 885,000 image-text pairs (sourced from CC3M, SBU, and LAION), the model trains a projection layer to align the visual encoder's outputs with the language model's embedding space, ensuring both components "speak the same language."

Stage 2: The Competitive Loop

The core of CoMD is an iterative three-phase cycle:

  1. Instruction Tuning: The student is trained on the teacher's responses to a diverse instruction pool (initially LLaVA-80K).
  2. Multi-modal Assessment: Both models answer the same set of questions. An assessment model scores the responses, and a difficulty formula identifies "Difficult Instructions"—tasks where the teacher succeeds but the student fails.
  3. Multi-modal Augmentation: An "Augmentor" model generates new, similarly challenging instructions based on the identified difficult tasks. These are added to the training set, while "easy" tasks are phased out to maintain curriculum difficulty.
The CoMD framework uses a bidirectional feedback loop to identify and augment difficult instructions by comparing student and teacher performance.

Through four iterations, the student masters a curriculum specifically tailored to its evolving weaknesses, eventually growing the instruction set to 504K samples.

Experimental Results

The CoMD 7B student model was evaluated against several benchmarks, demonstrating performance that frequently exceeded its 13B teacher:

  • ScienceQA: CoMD achieved 91.94% accuracy, surpassing the LLaVA-13B teacher (90.92%) and the previous state-of-the-art MM-CoT Large.
  • LLaVA Test Set: The 7B model scored 85.7, outperforming the 13B teacher (85.1) and significantly beating other 7B models like InstructBLIP.
  • SEED-Bench: In zero-shot evaluations, CoMD led most 7B-class open-source models in visual reasoning and text recognition.

Key Insights

  • The "Difficulty" Threshold: A threshold ($\tau$) of 0.33 was identified as the "sweet spot" for selecting difficult tasks. A curriculum that is too easy leads to stagnation, while one that is too difficult prevents effective learning.
  • Quality Over Scale: The results suggest that data relevance and quality are more critical than model size. A 7B model with a competitive curriculum can develop superior reasoning capabilities compared to a 13B model trained on static data.
  • Bidirectional Efficiency: Treating the student as an active participant that "flags" hard problems makes the distillation process much more sample-efficient.

Discussion and Limitations

While CoMD shows significant promise, some limitations remain. The model lags behind specialized architectures like InstructBLIP in spatial reasoning tasks (e.g., "Instance Location"), likely due to the latter's use of larger, task-specific datasets. Additionally, while CoMD reduces hallucinations compared to base LLaVA, it is not immune to them, occasionally misidentifying scene types based on misleading decorative elements (e.g., mislabeling a kitchen as a living room due to festive decorations).

Future work will focus on adapting the CoMD framework to other open-source architectures and exploring the performance ceiling of extended competitive iterations.

Summary updated

BibTeX
@misc{li2023unlock,
  author = {Xinwei Li and Li Lin and Shuai Wang and Chen Qian},
  title = {{Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models}},
  year = {2023},
  eprint = {2311.08213},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.08213},
}
Figure