Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition

Muhammad Haseeb Aslam, Marco Pedersoli, Alessandro Lameiras Koerich, Eric GrangerarXiv 2024 · DOI · arXiv

Multimodal Emotion Recognition (MER) has traditionally relied on the fusion of diverse data streams—facial video, audio, and physiological signals such as ECG or EMG—to achieve high accuracy. However, a significant gap exists between laboratory settings and real-world applications. In "in-the-wild" scenarios, such as driver monitoring or mobile health apps, medical-grade physiological sensors are rarely available. This creates a need for models that can leverage multimodal insights during training but operate using only a single modality (typically video) during inference.

The standard approach to this problem is Privileged Knowledge Distillation (PKD). In this paradigm, a "teacher" model trained on all modalities transfers its knowledge to a "student" model that only has access to the primary modality. Despite its promise, existing PKD methods suffer from two primary limitations: a reliance on rigid point-to-point feature matching and the "single teacher bottleneck," where the student is limited by the performance and potential biases of a single joint-modality teacher.

To address these challenges, the authors introduce MT-PKDOT (Multi-Teacher Privileged Knowledge Distillation with Optimal Transport). This framework shifts the focus from matching raw feature values to capturing the underlying structural relationships within the data.

The MT-PKDOT Framework

The MT-PKDOT architecture is built on three core innovations designed to make the student model more robust and capable of "hallucinating" the benefits of missing sensors.

1. Multi-Teacher Pool and Dynamic Selection

Rather than relying on a single teacher that fuses all modalities, MT-PKDOT utilizes a pool of teachers. This pool includes individual modality backbones and a joint fusion model. To ensure these diverse teachers can communicate with the student, the authors use Modality Adapters—encoder-decoder networks that align different feature spaces into a common representation. During training, the system dynamically selects the "most confident" teacher for each batch, preventing "negative transfer" where a teacher's specific error might otherwise be forced onto the student.

2. Structural KD via Optimal Transport (OT)

A key insight of this work is that the geometry of the feature space is often more informative than specific coordinate values. MT-PKDOT employs entropy-regularized Optimal Transport to minimize the distance between the distribution of relationships in the teacher and student spaces. By matching similarity matrices rather than individual samples, the student learns the "relational" knowledge—how different expressions or pain levels relate to one another—which is significantly more robust than raw feature matching.

3. Centroid Loss for Stability

Distillation processes can be inherently noisy. To stabilize learning, the authors introduced Centroid Loss, a constraint that minimizes the distance between the geometric centers (centroids) of the teacher and student representations. This acts as a spatial anchor, reducing variance and accelerating convergence during the training process.

Experimental Validation

The researchers evaluated MT-PKDOT on two rigorous benchmarks:

  1. Biovid Heat Pain Database: A dataset for pain estimation containing video and physiological signals (GSR, EMG, ECG).
  2. Affwild2: A massive "in-the-wild" dataset for continuous emotion prediction (Valence and Arousal) using video and audio.

The results demonstrate that MT-PKDOT consistently outperforms state-of-the-art privileged distillation methods. On the Biovid dataset, the method improved the visual-only baseline by 5.5% in accuracy, reaching 79.68%. This performance nearly matches the 81.30% "upper bound" achieved by the full multimodal teacher. On the Affwild2 dataset, the framework showed improvements of 3% for valence and 5% for arousal (measured by the Concordance Correlation Coefficient) over the visual-only baseline.

Conclusion and Future Directions

MT-PKDOT represents a significant step forward in making multimodal intelligence accessible in sensor-limited environments. By prioritizing relational structure and leveraging a diverse pool of teachers, the framework allows student models to effectively internalize the nuances of physiological data without requiring the sensors themselves at runtime.

While highly effective, the authors note that future iterations could benefit from explicit metrics to ensure teacher diversity and more sophisticated attention mechanisms for alignment. As the field moves toward more complex, multi-modal interactions, the principles of structural distillation and multi-teacher selection will likely play a crucial role in bridging the gap between the lab and the real world.

Summary updated

Abstract

Human emotion is a complex phenomenon conveyed and perceived through facial expressions, vocal tones, body language, and physiological signals. Multimodal emotion recognition systems can perform well because they can learn complementary and redundant semantic information from diverse sensors. In real-world scenarios, only a subset of the modalities employed for training may be available at test time. Learning privileged information allows a model to exploit data from additional modalities that are only available during training. SOTA methods for PKD have been proposed to distill information from a teacher model (with privileged modalities) to a student model (without privileged modalities). However, such PKD methods utilize point-to-point matching and do not explicitly capture the relational information. Recently, methods have been proposed to distill the structural information. However, PKD methods based on structural similarity are primarily confined to learning from a single joint teacher representation, which limits their robustness, accuracy, and ability to learn from diverse multimodal sources. In this paper, a multi-teacher PKD (MT-PKDOT) method with self-distillation is introduced to align diverse teacher representations before distilling them to the student. MT-PKDOT employs a structural similarity KD mechanism based on a regularized optimal transport (OT) for distillation. The proposed MT-PKDOT method was validated on the Affwild2 and Biovid datasets. Results indicate that our proposed method can outperform SOTA PKD methods. It improves the visual-only baseline on Biovid data by 5.5%. On the Affwild2 dataset, the proposed method improves 3% and 5% over the visual-only baseline for valence and arousal respectively. Allowing the student to learn from multiple diverse sources is shown to increase the accuracy and implicitly avoids negative transfer to the student model.
BibTeX
@misc{aslam2024multi,
  author = {Muhammad Haseeb Aslam and Marco Pedersoli and Alessandro Lameiras Koerich and Eric Granger},
  title = {{Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition}},
  year = {2024},
  eprint = {2408.09035},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.09035},
}
Figure