TL;DR
This paper introduces a novel "reject option" for speech emotion recognition (SER) systems. By allowing the system to abstain from predicting emotions it's uncertain about, they achieve significant improvements in the accuracy of the predictions it does make, effectively filtering out ambiguous speech segments.
What Problem Does This Paper Address?
Recognizing emotions in spontaneous speech is notoriously difficult. Unlike acted emotions, natural speech often contains subtle, mixed, or ambiguous emotional expressions. This ambiguity leads to low agreement even among human listeners and, consequently, lower performance for SER systems. When an SER system makes a prediction, it's crucial to know how reliable that prediction is. Current systems often provide a prediction even for highly uncertain cases, potentially leading to misinterpretations in human-computer interactions or behavioral studies. This paper tackles the problem of improving the reliability of SER by introducing a mechanism for the system to "opt-out" of making a prediction when it's not confident.
Methods
The core innovation is the implementation of a "reject option" within a deep neural network (DNN) for SER. This means the system can choose to not classify a given speech sample if it deems its prediction unreliable. Two main criteria were explored for deciding whether to accept or reject a sample:
- Threshold on Neuronal Activations (Criterion 1): This method uses the Selective Guaranteed Risk (SGR) algorithm. It applies a threshold to the softmax probabilities (or equivalently, the neuronal activations just before the softmax layer) of the DNN. The goal is to learn a rejection function that guarantees a desired error rate with high probability. Essentially, if the highest predicted probability is not sufficiently high, the sample is rejected.
- Difference Between Top Two Predictions (Criterion 2): This approach, inspired by work in automatic speech recognition, looks at the difference between the probabilities of the two most likely emotional classes. If this difference is small, it suggests ambiguity, and the sample is rejected.
Both criteria involve optimizing thresholds using a validation set to balance the trade-off between classification accuracy (F1-score) and the proportion of samples accepted (coverage). The DNN architecture itself is standard, employing dense layers with ReLU activations, dropout, and batch normalization.
Datasets
The primary dataset used is the MSP-Podcast corpus (version 1.4). This corpus consists of spontaneous speech samples from audio-sharing websites, segmented into speaking turns. It's rich in emotional content and has been annotated for emotions. The authors used a subset of this corpus, focusing on categorical emotions: happiness, anger, sadness, surprise, fear, disgust, contempt, and neutral states. They also created a five-class problem by removing some less frequent categories. The corpus was split into training, validation, and testing sets, aiming for speaker independence.
Key Insights and Ideas
- Rejecting ambiguity improves confidence: The fundamental idea is that by selectively rejecting samples where the system is uncertain, the remaining accepted samples are more likely to be correctly classified. This directly addresses the problem of low inter-evaluator agreement in emotional speech.
- Trade-off between coverage and performance: As expected, reducing the "coverage" (i.e., rejecting more samples) leads to higher F1-scores for the accepted samples. The paper quantifies this trade-off.
- Rejected samples are genuinely ambiguous: The analysis of inter-evaluator agreement shows that samples rejected by the system exhibit lower agreement among human annotators, validating the system's ability to identify uncertain cases.
- DNNs are well-suited for this task: The use of DNNs, which have shown superior performance in SER compared to older methods like SVMs, provides a strong foundation for implementing this reject option.
Key Results
- Significant F1-score gains: By rejecting 25% of the test data (achieving 75% coverage), the system achieved relative gains in F1-score of up to 25.71% for the five-class problem and 20.63% for the eight-class problem.
- Consistent improvement across criteria: Both proposed criteria for implementing the reject option showed performance improvements as coverage decreased.
- Hard labels generally perform better: Training with hard labels (consensus-based annotations) generally led to better F1-scores compared to soft labels.
- Rejected samples have lower inter-evaluator agreement: Table 2 clearly shows that Fleiss' Kappa (a measure of agreement) is consistently lower for rejected samples compared to accepted samples across different coverage levels.
Limitations and Future Work
- Focus on categorical emotions: The current work focuses on discrete emotion categories. The authors suggest extending this framework to regression problems that predict emotional attributes like arousal, valence, and dominance.
- Exploring alternative criteria: While the two proposed criteria were effective, the authors mention exploring other criteria to further optimize the F1-score while maintaining high coverage.
- Real-world implications: The paper highlights the potential for this approach to improve the precision and efficiency of SER models for practical applications in behavioral and psychological studies.
Summary updated