Modality-specific Distillation

Woojeong Jin, Maziar Sanjabi, Shaoliang Nie, Liang Tan, Xiang Ren, Hamed FiroozNAACL 2021

TL;DR

This paper introduces Modality-Specific Distillation (MSD), a novel technique to improve knowledge distillation for multimodal AI models. MSD helps smaller "student" models learn more effectively from larger "teacher" models by focusing on each data modality (like text and images) separately, leading to better performance and efficiency.

What Problem Does This Paper Address?

Modern AI models, especially those dealing with multiple data types like text and images (multimodal AI), have become incredibly powerful but also enormous. This size makes them impractical for deployment on devices with limited computational resources, such as smartphones. Knowledge Distillation (KD) is a common technique to shrink these large "teacher" models into smaller "student" models while retaining performance. However, existing KD methods often struggle with multimodal data because they treat the combined input as a single entity, failing to account for the unique information each modality provides. This can lead to a "prediction gap" where the student doesn't fully capture the teacher's nuanced understanding of each individual modality, hindering effective knowledge transfer.

Methods

The core innovation is Modality-Specific Distillation (MSD). Instead of feeding the combined multimodal input to both teacher and student, MSD splits the input into its constituent modalities (e.g., text and image). Then, it trains the student to mimic the teacher's predictions for each modality separately. This is achieved by introducing auxiliary loss terms for each modality's distillation.

To further refine this process, the authors propose several weighting schemes for these auxiliary losses:

  • Population-based Weighting: Assigns fixed weights to each modality's loss term, assuming a general importance for each modality across the dataset.
  • Instance-wise Weighting: Dynamically adjusts the weights for each loss term based on individual data samples. This is further broken down into:
  • Importance-based Weighting: Weights are determined by how much dropping one modality affects the teacher's prediction.
  • Correctness-based Weighting: Weights are based on how accurate the teacher's prediction is for each modality.
  • Meta-Learning for Weights: The most advanced approach, where a separate "meta-learner" network is trained to find optimal, sample-specific weights for the distillation losses. This meta-learner learns to adapt the weighting strategy based on the characteristics of the data.

Datasets

The proposed methods were evaluated on three multimodal datasets:

  • Hateful-Memes: A dataset of 10K multimodal memes for binary classification (detecting hate speech).
  • MM-IMDB: A dataset of 26K movie plot outlines and posters for multi-label genre prediction.
  • Visual Entailment (SNLI-VE): A dataset for predicting whether an image semantically entails a given sentence, with three classes ("Entailment," "Neutral," "Contradiction").

Key Insights and Ideas

  • Modality Matters: The paper strongly suggests that treating multimodal data as a single unit in KD is suboptimal. Explicitly distilling knowledge from each modality separately is crucial for effective transfer.
  • Dynamic Weighting is Key: While fixed population-based weights offer improvements, instance-wise and meta-learned weights provide more significant gains, indicating that the importance of each modality can vary greatly from one data sample to another.
  • Meta-Learning Adapts Best: The meta-learning approach for weight optimization proved to be the most effective, demonstrating its ability to find dataset-specific, sample-aware weighting strategies that significantly boost performance.
  • Faster Learning: MSD, especially with meta-learning, leads to faster convergence during training compared to standard KD, suggesting the student learns more efficiently.

Key Results

  • MSD consistently outperforms conventional KD across all tested datasets and metrics.
  • The meta-learning weighting scheme for MSD achieved the best performance, often coming closest to the teacher model's performance.
  • For example, on the Hateful-Memes dataset, MSD with meta-learning achieved an AUC of 68.30%, compared to conventional KD's 66.53% and the small model's 65.54%.
  • MSD also showed significant improvements when applied on top of other existing KD approaches (Table 2).
  • The paper demonstrates that MSD improves student model performance regardless of student model size (number of layers).

Limitations and Future Work

  • Dataset Bias: The paper acknowledges that different datasets might benefit more from population-based versus instance-wise weighting, suggesting that the optimal weighting strategy is dataset-dependent.
  • Generalizability of Meta-Learner: While the meta-learning approach is flexible, its effectiveness might depend on the quality and size of the meta-data set (validation set in this case).
  • Extending to More Modalities: The current work focuses on text and image modalities. Future work could explore extending MSD to datasets with more than two modalities.
  • Investigating Weighting Functions: While meta-learning finds optimal weights, further research into designing more intuitive and efficient weighting functions could be beneficial.

Summary updated

BibTeX
@inproceedings{jin2021modality,
  author = {Woojeong Jin and Maziar Sanjabi and Shaoliang Nie and Liang Tan and Xiang Ren and Hamed Firooz},
  title = {{Modality-specific Distillation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure