VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

Yanan Wang, Donghuo Zeng, Shinya Wada, Satoshi Kurihara

Abstract

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference.

Introduction

In the field of artificial intelligence, multimodal models—those capable of processing visual, acoustic, and textual information—represent the state-of-the-art for complex tasks such as sentiment analysis. Despite their performance, these systems face significant challenges in real-world deployment. First, they exhibit inference rigidity, assuming all input modalities are available at runtime. If a modality like video is missing, performance typically degrades sharply. Second, the computational cost of processing high-dimensional video data is prohibitive for real-time applications.

VideoAdviser addresses this "efficiency-performance" trade-off by enabling a text-only model to inherit the "visual imagination" of a multimodal teacher during training, allowing for efficient, high-performance inference using only text.

Comparison between a conventional multimodal system and the proposed VideoAdviser framework.

Methodology: The VideoAdviser Framework

The core of VideoAdviser is a teacher-student architecture designed to transfer rich multimodal knowledge into a unimodal student.

The Teacher (CLIP-based)

The teacher model utilizes CLIP, enhanced by a facial expression encoder (OpenFace) and a video-specific prompting module. Rather than simply processing raw frames, it uses "video-enhanced prompts" (e.g., "A video with a {positive} face") to bridge the gap between visual features and linguistic labels, providing a more expressive supervision signal.

The Student (RoBERTa-based)

The student is a standard RoBERTa-Large model. While it learns from the multimodal teacher during training, it only requires text input during inference, eliminating the need for video processing at runtime.

Two-Step Distillation

The framework employs a unique step-distillation objective:

  1. Step 1: The teacher distills its own coarse classification knowledge (e.g., sentiment categories) into a fine-grained regression logit. This step unifies multimodal knowledge across different granularities.
  2. Step 2: This unified knowledge is then distilled from the teacher’s regression logit to the student’s text logit.

Experimental Evaluation

The researchers evaluated VideoAdviser on two distinct tasks:

  • Video-level Sentiment Analysis: Using the MOSI and MOSEI datasets, which contain video clips annotated with sentiment scores ranging from -3 to +3.
  • Audio-Visual Retrieval: Using the VEGAS dataset, featuring over 28,000 videos categorized by audio events (e.g., "baby crying," "fireworks").

Key Results

VideoAdviser demonstrated significant improvements over existing state-of-the-art (SOTA) methods:

  • Sentiment Analysis: The student model achieved a Mean Absolute Error (MAE) improvement of up to 12.3% on MOSI and MOSEI.
  • Efficiency: A VideoAdviser student using RoBERTa-Base outperformed the SOTA UniMSE model while requiring nearly half the parameters.
  • Retrieval: On the VEGAS dataset, the method enhanced the mean Average Precision (mAP) score by 3.4% without any additional inference-time computation.

Discussion and Conclusion

The success of VideoAdviser stems from its ability to provide "professional advice" from a powerful multimodal teacher to a "smart student." By forcing the teacher to align classification and regression outputs before distillation, the student captures the intensity of sentiment more effectively. While the method requires full finetuning of the student model to be effective, it offers a powerful solution for deploying multimodal-aware models in resource-constrained environments. By giving text models a "visual imagination," VideoAdviser achieves a superior balance of efficiency and performance in multimodal transfer learning.

Summary updated

Abstract

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). With an intuition that the best learning performance comes with professional advisers and smart students, we use a CLIP-based teacher model to provide expressive multimodal knowledge supervision signals to a RoBERTa-based student model via optimizing a step-distillation objective loss—first step: the teacher distills multimodal knowledge of video-enhanced prompts from classification logits to a regression logit—second step: the multimodal knowledge is distilled from the regression logit of the teacher to the student. We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference. These results suggest the strengths of our method for achieving high efficiency-performance multimodal transfer learning.
BibTeX
@inproceedings{wang2023videoadviser,
  author = {Yanan Wang and Donghuo Zeng and Shinya Wada and Satoshi Kurihara},
  title = {{VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=u72VSt9RK2},
}
Figure