Abstract
Deep Metric Learning (DML) typically relies on discrete "hard" labels, which fails to capture the inherent subjectivity of tasks like Speech Emotion Recognition (SER). We present $f$-Similarity Preservation Loss ($f$-SPL), a novel family of loss functions derived from $f$-divergence that enables DML to utilize "soft" continuous labels. By enforcing the preservation of pairwise label similarities in the embedding space, $f$-SPL produces more robust representations. Our experiments on cross-corpus SER tasks using the IEMOCAP and MSP-Improv datasets demonstrate that $f$-SPL significantly improves generalization and prevents performance degradation during over-training compared to standard cross-entropy baselines.
Introduction
Deep Metric Learning (DML) aims to map input data into an embedding space where similar items are clustered and dissimilar items are separated. While effective for categorical data (e.g., "Cat" vs. "Dog"), standard DML struggles with subjective domains like Speech Emotion Recognition (SER). In SER, an utterance rarely belongs to a single discrete category; instead, it is often characterized by "soft labels" reflecting a distribution of annotator perceptions. Furthermore, SER models frequently suffer from poor cross-corpus generalization, overfitting to dataset-specific artifacts rather than underlying emotional features.
Methodology
The core contribution of this work is the $f$-Similarity Preservation Loss ($f$-SPL). Unlike traditional triplet losses that seek only to separate discrete classes, $f$-SPL ensures that the similarity between two samples in the learned embedding space directly reflects the similarity of their soft labels.
Framework and Sampling
The model utilizes a multi-task framework, combining a standard cross-entropy classification loss with the $f$-SPL term. To satisfy the theoretical requirements for similarity preservation, the authors implemented a Balanced Triplet Sampling strategy. For every anchor sample, a similar and a dissimilar sample are selected such that their similarities to the anchor balance around a neutral point.
By varying the convex function ($f$) used in the underlying $f$-divergence, a family of losses can be generated, including versions based on KL-divergence, Pearson $\chi^2$, and Jensen-Shannon divergence.
Experimental Setup
The approach was evaluated using a strict cross-corpus protocol between two major datasets:
- IEMOCAP: 12 hours of dyadic interactions.
- MSP-Improv: 9 hours of emotional speech in varied contexts.
Models were trained on one corpus and tested on the other to assess their ability to generalize across different speakers and recording conditions.
Results and Discussion
Performance and Stability
In most cross-corpus scenarios, $f$-SPL models significantly outperformed the baseline Cross-Entropy (CE) model in Unweighted Average Recall (UAR). A critical finding was the stability of $f$-SPL over time. While the CE-only baseline typically reached a peak and then suffered from performance degradation due to over-training, $f$-SPL variants maintained high performance across many epochs.
Embedding Space Characteristics
Visualization of the embedding space revealed that $f$-SPL creates a continuous "emotional gradient." Samples with ambiguous or neutral soft labels (e.g., [0.5, 0.5]) naturally reside between clear emotional clusters, effectively treating soft labels as informative features rather than noise.
Conclusion
$f$-SPL provides a principled way to incorporate subjective, soft-labeled data into Deep Metric Learning. By preserving label similarities in the embedding space, the framework enhances model robustness and cross-corpus generalization in SER. Future work will explore the application of $f$-SPL to transfer learning scenarios with limited target-domain data.
Summary updated