Abstract
Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis and audio-visual retrieval. The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference.
Introduction
Multimodal learning, which integrates information from text, video, and audio, has become a cornerstone of modern AI applications. However, two significant hurdles persist: the "missing modality" problem and the high computational cost of inference. Most existing frameworks assume that all data streams (e.g., video frames and audio tracks) are available during deployment. If a modality is missing or corrupted, performance degrades sharply. Moreover, running multiple heavy encoders—such as a Vision Transformer for video and a Large Language Model for text—is often too slow for real-time applications.
VideoAdviser addresses these challenges by leveraging knowledge distillation. The goal is to train a model that "sees" the video during training but can perform effectively using only text during inference. By transferring rich multimodal insights from a "teacher" model to a "student" model, VideoAdviser achieves high performance with minimal computational overhead.
The VideoAdviser Framework
The core of VideoAdviser is a teacher-student architecture designed for efficient multimodal transfer.
1. The Professional Teacher
The teacher model is built upon a CLIP-based architecture. To transform it into a multimodal expert, the researchers augment it with a facial expression encoder and a video-specific prompting module. This allows the teacher to capture both high-level semantic context and fine-grained emotional cues from video frames, creating a robust "video-enhanced" representation.
2. The Smart Student
The student model is a RoBERTa-based language model. Unlike traditional fusion models, the student only requires text input during inference. This design choice ensures that the system remains lightweight and resilient to missing visual or audio data in production environments.
3. Step-Distillation: The Learning Objective
The primary innovation of VideoAdviser is its "step-distillation" process. Rather than a direct transfer of knowledge, which can be noisy, the framework employs a two-stage approach:
- Step 1 (Internal Teacher Alignment): The teacher distills its own internal knowledge from coarse-grained classification logits (e.g., discrete sentiment labels) into fine-grained regression logits (e.g., continuous sentiment scores). This step ensures the teacher's multimodal knowledge is unified and precise.
- Step 2 (Teacher-to-Student Transfer): This refined multimodal knowledge is then distilled from the teacher’s regression logit into the student’s text logit.
By optimizing this objective, the student learns to represent text in a way that accounts for the visual nuances it "observed" via the teacher during the training phase.
Experimental Evaluation
The researchers evaluated VideoAdviser on three benchmark datasets representing distinct multimodal challenges:
- MOSI & MOSEI: These are the gold standards for video-level sentiment analysis. They consist of YouTube monologues where speakers express various opinions, annotated with sentiment scores ranging from highly negative to highly positive.
- VEGAS: A dataset used for audio-visual retrieval tasks. It contains over 28,000 videos across 10 audio event categories, such as "baby crying" or "fireworks," requiring the model to associate visual scenes with specific acoustic signatures.
Results and Impact
VideoAdviser demonstrates that a text-only model can outperform complex multimodal fusion models if it is "advised" correctly during training.
Performance Gains
On the MOSI and MOSEI datasets, the student model achieved a 12.3% improvement in Mean Absolute Error (MAE) compared to previous state-of-the-art (SoTA) multimodal models. In retrieval tasks using the VEGAS dataset, the method improved the mean Average Precision (mAP) by 3.4% over existing SoTA benchmarks.
Efficiency and Scalability
One of the most significant findings is the efficiency of the student model. A RoBERTa-Base student trained via VideoAdviser outperformed much larger models, such as UniMSE, while utilizing nearly half the parameters. Because the teacher model is only required during the training phase, the inference process is significantly faster and requires no additional computation for video or audio processing.
Conclusion
VideoAdviser presents a compelling case for "multimodal knowledge without multimodal inference." By using video-enhanced prompts and a unique two-step distillation loss, the framework allows lightweight text models to inherit the depth of multimodal understanding. This approach not only sets new performance benchmarks in sentiment analysis and retrieval but also provides a practical path forward for deploying sophisticated AI in resource-constrained or data-incomplete environments. Future work may explore meta-learning techniques to adapt this distillation process for few-shot scenarios.
Summary updated