Introduction
Large-scale multimodal foundation models, such as CoCa and CLIP, have set new benchmarks across various vision-language tasks. However, their massive parameter counts make them computationally prohibitive for real-time deployment. Knowledge distillation, specifically layerwise distillation (LWD), is a common technique to compress these "teacher" models into smaller "student" models.
Traditional LWD approaches typically treat all architectural modules—such as image encoders, text encoders, and multimodal decoders—with equal importance throughout the training process. This research identifies a significant inefficiency in this uniform approach: different tasks rely on different modules, and distilling irrelevant components can lead to gradient interference, hindering the student's learning efficiency.
The OPTIMA Framework
To address these limitations, the authors introduce OPTIMA (Module-wise Adaptive Distillation with Multi-arm Bandit). OPTIMA treats the selection of modules for distillation as a non-stationary Multi-Armed Bandit (MAB) problem.
In this framework, each "arm" represents a specific module or a combination of modules. The "reward" for selecting an arm is defined by the reduction in distillation loss. Because the relative importance of modules shifts as the student model matures, OPTIMA employs a modified Thompson Sampling algorithm integrated with an Exponential Moving Average (EMA). This allows the system to adaptively prioritize modules that offer the highest performance gains at any given stage of training.
Task-Specific Dynamics and Interference
A key insight of this work is that module importance is highly task-dependent. For instance, in image captioning tasks (e.g., COCO Caption), the image encoder is the primary driver of performance. Conversely, for reasoning tasks like NLVR2, the multimodal decoder plays a more critical role.
Standard distillation methods that apply uniform weights across all modules often underperform compared to targeted distillation. By adaptively selecting the most relevant modules, OPTIMA avoids the "interference" caused by training on less critical components, leading to more robust student models.
Experimental Results
The researchers evaluated OPTIMA by distilling a CoCa-Large teacher (672M parameters) into CoCa-Tiny students (55M and 102M parameters). The benchmarks included VQA v2.0, SNLI-VE, NLVR2, and Microsoft COCO Caption.
- Performance Gains: OPTIMA consistently outperformed standard LWD baselines. On the COCO Captioning benchmark, OPTIMA achieved a CIDEr score of 121.2, compared to 116.8 for standard LWD.
- Model Efficiency: Remarkably, a 102M-parameter student trained via OPTIMA outperformed the 293M-parameter CoCa-Base model on three out of four benchmarks, demonstrating that adaptive distillation can produce highly efficient models that punch above their weight class.
- Generalization: Student models trained with OPTIMA showed lower prediction loss and improved output discrepancy scores, indicating superior generalization to unseen data compared to fixed-distillation strategies.
Conclusion and Future Directions
OPTIMA demonstrates that adaptive, module-wise distillation is a powerful strategy for compressing multimodal foundation models. By framing module selection as a bandit problem, the framework navigates the complex trade-offs between different modalities and tasks.
Future work may explore finer-grained module selection (e.g., specific layer groups) and investigate the dependencies between coupled modules using more advanced reinforcement learning techniques. While currently focused on vision-language models, the OPTIMA framework is potentially applicable to a wide range of multimodal architectures, including those involving audio and video.
Summary updated