VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

Yanan Wang, Donghuo Zeng, Shinya Wada, Satoshi Kurihara

Abstract

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis and audio-visual retrieval. The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference.

Introduction

Multimodal learning, which integrates information from text, video, and audio, has become a cornerstone of modern AI applications. However, two significant hurdles persist: the "missing modality" problem and the high computational cost of inference. Most existing frameworks assume that all data streams (e.g., video frames and audio tracks) are available during deployment. If a modality is missing or corrupted, performance degrades sharply. Moreover, running multiple heavy encoders—such as a Vision Transformer for video and a Large Language Model for text—is often too slow for real-time applications.

VideoAdviser addresses these challenges by leveraging knowledge distillation. The goal is to train a model that "sees" the video during training but can perform effectively using only text during inference. By transferring rich multimodal insights from a "teacher" model to a "student" model, VideoAdviser achieves high performance with minimal computational overhead.

The VideoAdviser Framework

The core of VideoAdviser is a teacher-student architecture designed for efficient multimodal transfer.

1. The Professional Teacher

The teacher model is built upon a CLIP-based architecture. To transform it into a multimodal expert, the researchers augment it with a facial expression encoder and a video-specific prompting module. This allows the teacher to capture both high-level semantic context and fine-grained emotional cues from video frames, creating a robust "video-enhanced" representation.

2. The Smart Student

The student model is a RoBERTa-based language model. Unlike traditional fusion models, the student only requires text input during inference. This design choice ensures that the system remains lightweight and resilient to missing visual or audio data in production environments.

3. Step-Distillation: The Learning Objective

The primary innovation of VideoAdviser is its "step-distillation" process. Rather than a direct transfer of knowledge, which can be noisy, the framework employs a two-stage approach:

  • Step 1 (Internal Teacher Alignment): The teacher distills its own internal knowledge from coarse-grained classification logits (e.g., discrete sentiment labels) into fine-grained regression logits (e.g., continuous sentiment scores). This step ensures the teacher's multimodal knowledge is unified and precise.
  • Step 2 (Teacher-to-Student Transfer): This refined multimodal knowledge is then distilled from the teacher’s regression logit into the student’s text logit.

By optimizing this objective, the student learns to represent text in a way that accounts for the visual nuances it "observed" via the teacher during the training phase.

Experimental Evaluation

The researchers evaluated VideoAdviser on three benchmark datasets representing distinct multimodal challenges:

  • MOSI & MOSEI: These are the gold standards for video-level sentiment analysis. They consist of YouTube monologues where speakers express various opinions, annotated with sentiment scores ranging from highly negative to highly positive.
  • VEGAS: A dataset used for audio-visual retrieval tasks. It contains over 28,000 videos across 10 audio event categories, such as "baby crying" or "fireworks," requiring the model to associate visual scenes with specific acoustic signatures.

Results and Impact

VideoAdviser demonstrates that a text-only model can outperform complex multimodal fusion models if it is "advised" correctly during training.

Performance Gains

On the MOSI and MOSEI datasets, the student model achieved a 12.3% improvement in Mean Absolute Error (MAE) compared to previous state-of-the-art (SoTA) multimodal models. In retrieval tasks using the VEGAS dataset, the method improved the mean Average Precision (mAP) by 3.4% over existing SoTA benchmarks.

Efficiency and Scalability

One of the most significant findings is the efficiency of the student model. A RoBERTa-Base student trained via VideoAdviser outperformed much larger models, such as UniMSE, while utilizing nearly half the parameters. Because the teacher model is only required during the training phase, the inference process is significantly faster and requires no additional computation for video or audio processing.

Conclusion

VideoAdviser presents a compelling case for "multimodal knowledge without multimodal inference." By using video-enhanced prompts and a unique two-step distillation loss, the framework allows lightweight text models to inherit the depth of multimodal understanding. This approach not only sets new performance benchmarks in sentiment analysis and retrieval but also provides a practical path forward for deploying sophisticated AI in resource-constrained or data-incomplete environments. Future work may explore meta-learning techniques to adapt this distillation process for few-shot scenarios.

Summary updated

Abstract

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). With an intuition that the best learning performance comes with professional advisers and smart students, we use a CLIP-based teacher model to provide expressive multimodal knowledge supervision signals to a RoBERTa-based student model via optimizing a step-distillation objective loss -- first step: the teacher distills multimodal knowledge of video-enhanced prompts from classification logits to a regression logit -- second step: the multimodal knowledge is distilled from the regression logit of the teacher to the student. We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference. These results suggest the strengths of our method for achieving high efficiency-performance multimodal transfer learning.
BibTeX
@inproceedings{wang2023videoadviser,
  author = {Yanan Wang and Donghuo Zeng and Shinya Wada and Satoshi Kurihara},
  title = {{VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=u72VSt9RK2},
}
Figure