Multimodal Emotion Recognition (MER) has traditionally relied on the fusion of diverse data streams—facial video, audio, and physiological signals such as ECG or EMG—to achieve high accuracy. However, a significant gap exists between laboratory settings and real-world applications. In "in-the-wild" scenarios, such as driver monitoring or mobile health apps, medical-grade physiological sensors are rarely available. This creates a need for models that can leverage multimodal insights during training but operate using only a single modality (typically video) during inference.
The standard approach to this problem is Privileged Knowledge Distillation (PKD). In this paradigm, a "teacher" model trained on all modalities transfers its knowledge to a "student" model that only has access to the primary modality. Despite its promise, existing PKD methods suffer from two primary limitations: a reliance on rigid point-to-point feature matching and the "single teacher bottleneck," where the student is limited by the performance and potential biases of a single joint-modality teacher.
To address these challenges, the authors introduce MT-PKDOT (Multi-Teacher Privileged Knowledge Distillation with Optimal Transport). This framework shifts the focus from matching raw feature values to capturing the underlying structural relationships within the data.
The MT-PKDOT Framework
The MT-PKDOT architecture is built on three core innovations designed to make the student model more robust and capable of "hallucinating" the benefits of missing sensors.
1. Multi-Teacher Pool and Dynamic Selection
Rather than relying on a single teacher that fuses all modalities, MT-PKDOT utilizes a pool of teachers. This pool includes individual modality backbones and a joint fusion model. To ensure these diverse teachers can communicate with the student, the authors use Modality Adapters—encoder-decoder networks that align different feature spaces into a common representation. During training, the system dynamically selects the "most confident" teacher for each batch, preventing "negative transfer" where a teacher's specific error might otherwise be forced onto the student.
2. Structural KD via Optimal Transport (OT)
A key insight of this work is that the geometry of the feature space is often more informative than specific coordinate values. MT-PKDOT employs entropy-regularized Optimal Transport to minimize the distance between the distribution of relationships in the teacher and student spaces. By matching similarity matrices rather than individual samples, the student learns the "relational" knowledge—how different expressions or pain levels relate to one another—which is significantly more robust than raw feature matching.
3. Centroid Loss for Stability
Distillation processes can be inherently noisy. To stabilize learning, the authors introduced Centroid Loss, a constraint that minimizes the distance between the geometric centers (centroids) of the teacher and student representations. This acts as a spatial anchor, reducing variance and accelerating convergence during the training process.
Experimental Validation
The researchers evaluated MT-PKDOT on two rigorous benchmarks:
- Biovid Heat Pain Database: A dataset for pain estimation containing video and physiological signals (GSR, EMG, ECG).
- Affwild2: A massive "in-the-wild" dataset for continuous emotion prediction (Valence and Arousal) using video and audio.
The results demonstrate that MT-PKDOT consistently outperforms state-of-the-art privileged distillation methods. On the Biovid dataset, the method improved the visual-only baseline by 5.5% in accuracy, reaching 79.68%. This performance nearly matches the 81.30% "upper bound" achieved by the full multimodal teacher. On the Affwild2 dataset, the framework showed improvements of 3% for valence and 5% for arousal (measured by the Concordance Correlation Coefficient) over the visual-only baseline.
Conclusion and Future Directions
MT-PKDOT represents a significant step forward in making multimodal intelligence accessible in sensor-limited environments. By prioritizing relational structure and leveraging a diverse pool of teachers, the framework allows student models to effectively internalize the nuances of physiological data without requiring the sensors themselves at runtime.
While highly effective, the authors note that future iterations could benefit from explicit metrics to ensure teacher diversity and more sophisticated attention mechanisms for alignment. As the field moves toward more complex, multi-modal interactions, the principles of structural distillation and multi-teacher selection will likely play a crucial role in bridging the gap between the lab and the real world.
Summary updated