TL;DR
This paper introduces a novel method for estimating the confidence of speech emotion recognition (SER) system predictions. By leveraging semi-supervised learning across multiple emotion corpora, the authors developed a system that can reliably indicate how much trust to place in a given emotion classification, a crucial step for real-world SER applications.
What Problem Does This Paper Address?
Speech emotion recognition (SER) systems are becoming more accurate, but their decisions aren't always perfect. This is due to various factors like noisy audio, differences between acted and natural emotions, and limitations in the data itself. For SER to be useful in real-world scenarios (e.g., in human-computer interaction or mental health monitoring), it's essential for the system to not only predict an emotion but also to signal how confident it is in that prediction. Existing confidence measures are largely designed for Automatic Speech Recognition (ASR) and rely on components (like Hidden Markov Models and word graphs) not typically found in SER systems. This paper aims to bridge this gap by proposing a confidence measure specifically for SER.
Methods
The core of this paper is a semi-supervised learning approach to build confidence measures (CMs). The process involves:
- Multi-Corpus Training: The authors use five different speech emotion corpora as "teachers." Each corpus is used to train a separate classifier.
- Iterative Refinement: An iterative self-training procedure is employed. In each iteration, the unlabeled training data for the primary SER task (FAU Aibo corpus) is predicted by the ensemble of trained classifiers.
- High-Confidence Instance Selection: Instances where the majority of the "teacher" classifiers agree are considered "high confidence." These instances are then added to the training set of the respective classifiers, effectively reinforcing their learning.
- Confidence Ratio Calculation: After the iterative training, the authors calculate four types of "confidence ratios" based on the confusion matrices of the trained classifiers: Correct Negative (CN), Incorrect Negative (ICN), Correct Positive (CP), and Incorrect Positive (ICP).
- Final Confidence Measure (SCM): These individual confidence ratios are then combined, using a weighted average, to produce a single, normalized confidence score (SCM) for each prediction. This SCM ranges from 0 to 1, indicating the system's confidence in its prediction.
Datasets
The study utilizes six emotion corpora:
- Training/Evaluation: FAU Aibo emotion corpus (German, spontaneous speech) - used as the primary dataset for training the SER system and evaluating the proposed CM.
- "Teacher" Corpora for Semi-Supervised Learning:
- Airplane Behaviour Corpus (ABC) (German, acted speech)
- Audiovisual Interest Corpus (AVIC) (English, natural speech)
- Danish Emotional Speech (DES) (Danish, acted speech)
- ENTERFACE database (eNTER) (English, induced speech)
- Belfast Sensitive Artificial Listener (SAL) (English, natural speech)
All corpora were mapped to a binary valence axis (positive/negative) for consistency.
Key Insights and Ideas
- Leveraging Multiple Corpora for Confidence: The key innovation is using diverse emotion corpora not just for training the primary SER system, but as a collective "wisdom" to train confidence estimators. This ensemble approach helps overcome the limitations of any single dataset.
- Semi-Supervised Learning for Confidence: The iterative self-training process effectively bootstraps confidence estimation by gradually incorporating high-confidence predictions into the training data, leading to more robust confidence scores.
- Confidence as a Reliability Indicator: The proposed SCM provides a quantifiable measure of how likely a prediction is correct, which is crucial for deploying SER systems in applications where errors can have consequences.
- Beyond ASR: The paper successfully adapts confidence estimation techniques to the SER domain, addressing a gap in existing research.
Key Results
- The proposed semi-supervised learning method for confidence measures showed significant improvement over random guessing.
- In Receiver Operating Characteristic (ROC) space, the method achieved a maximum Area Under the Curve (AUC) of 66.43% at iteration 3, substantially higher than the random guess AUC of 50%.
- The Normalized Cross Entropy (NCE) for the proposed method on the FAU Aibo corpus was 0.80, which is close to the optimal value of 1, indicating effective confidence estimation.
Limitations and Future Work
- SER System Accuracy: The paper notes that the unweighted and weighted accuracies of the underlying SER system on the FAU Aibo dataset were 68.7% and 69.2%, respectively. While the confidence measure is evaluated, its effectiveness is inherently tied to the base SER system's performance.
- Future Work: The authors express interest in using these confidence measures to further enhance semi-supervised learning on unlabeled emotional speech data. They also suggest exploring other classification models for confidence estimation.
Summary updated