Abstract
Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference.
Introduction
In the field of artificial intelligence, multimodal models—those capable of processing visual, acoustic, and textual information—represent the state-of-the-art for complex tasks such as sentiment analysis. Despite their performance, these systems face significant challenges in real-world deployment. First, they exhibit inference rigidity, assuming all input modalities are available at runtime. If a modality like video is missing, performance typically degrades sharply. Second, the computational cost of processing high-dimensional video data is prohibitive for real-time applications.
VideoAdviser addresses this "efficiency-performance" trade-off by enabling a text-only model to inherit the "visual imagination" of a multimodal teacher during training, allowing for efficient, high-performance inference using only text.
Methodology: The VideoAdviser Framework
The core of VideoAdviser is a teacher-student architecture designed to transfer rich multimodal knowledge into a unimodal student.
The Teacher (CLIP-based)
The teacher model utilizes CLIP, enhanced by a facial expression encoder (OpenFace) and a video-specific prompting module. Rather than simply processing raw frames, it uses "video-enhanced prompts" (e.g., "A video with a {positive} face") to bridge the gap between visual features and linguistic labels, providing a more expressive supervision signal.
The Student (RoBERTa-based)
The student is a standard RoBERTa-Large model. While it learns from the multimodal teacher during training, it only requires text input during inference, eliminating the need for video processing at runtime.
Two-Step Distillation
The framework employs a unique step-distillation objective:
- Step 1: The teacher distills its own coarse classification knowledge (e.g., sentiment categories) into a fine-grained regression logit. This step unifies multimodal knowledge across different granularities.
- Step 2: This unified knowledge is then distilled from the teacher’s regression logit to the student’s text logit.
Experimental Evaluation
The researchers evaluated VideoAdviser on two distinct tasks:
- Video-level Sentiment Analysis: Using the MOSI and MOSEI datasets, which contain video clips annotated with sentiment scores ranging from -3 to +3.
- Audio-Visual Retrieval: Using the VEGAS dataset, featuring over 28,000 videos categorized by audio events (e.g., "baby crying," "fireworks").
Key Results
VideoAdviser demonstrated significant improvements over existing state-of-the-art (SOTA) methods:
- Sentiment Analysis: The student model achieved a Mean Absolute Error (MAE) improvement of up to 12.3% on MOSI and MOSEI.
- Efficiency: A VideoAdviser student using RoBERTa-Base outperformed the SOTA UniMSE model while requiring nearly half the parameters.
- Retrieval: On the VEGAS dataset, the method enhanced the mean Average Precision (mAP) score by 3.4% without any additional inference-time computation.
Discussion and Conclusion
The success of VideoAdviser stems from its ability to provide "professional advice" from a powerful multimodal teacher to a "smart student." By forcing the teacher to align classification and regression outputs before distillation, the student captures the intensity of sentiment more effectively. While the method requires full finetuning of the student model to be effective, it offers a powerful solution for deploying multimodal-aware models in resource-constrained environments. By giving text models a "visual imagination," VideoAdviser achieves a superior balance of efficiency and performance in multimodal transfer learning.
Summary updated