TL;DR
Explainable Speech Emotion Recognition (SER) relies on predicting Speech Emotion Descriptors (SEDs) like pitch and speaker traits, but automatically generated training labels for these descriptors are notoriously noisy. This paper introduces a dual-mechanism approach combining a confidence-based data filtering module and a reinforcement learning-driven "on-the-fly" label rectifier. This framework significantly boosts both the classification accuracy and the trustworthiness of explainable emotion recognition models.
What Problem Does This Paper Address?
Traditional Speech Emotion Recognition (SER) systems act as black boxes, classifying an audio clip into a discrete category (e.g., "angry" or "sad") without explaining why. Recent Speech Large Language Models (SLMs) have introduced explainability by jointly predicting the emotion alongside Speech Emotion Descriptors (SEDs)—such as pitch, volume, speed, age, and gender.
However, high-quality, human-annotated SED data is incredibly scarce. To scale training, researchers rely on automated tools with fixed acoustic thresholds to label data. This introduces massive noise because a "high pitch" or "loud volume" varies drastically across different ages, genders, and accents. Training on this unreliable data causes a trust issue: a model might guess the correct emotion but output completely incorrect acoustic descriptions, rendering its "explanation" useless.
Methods
The authors propose a post-training pipeline featuring two core components designed to clean up and rectify noisy automated SED labels:
-
Confidence Score-based Data Selection: The authors build a lightweight Multi-Layer Perceptron (MLP)-based Confidence Estimation Module (CEM). By feeding the hidden states from a pre-trained SER-SLM into this module, it calculates a smoothed confidence score for each utterance's automated SED labels. This allows the system to filter out the most unreliable data before fine-tuning.
-
RL-based On-the-Fly SED Rectification: Filtering data too aggressively discards difficult but valuable training samples. To counteract this, they introduce an Recurrent Neural Network (RNN)-based SED Controller. Operating as a reinforcement learning agent during SLM training, the controller generates multiple "rectification policies" (deciding whether to retain or modify a noisy SED label).
The reward mechanism is the clever bit: the controller is rewarded if its modified SED labels result in a lower cross-entropy loss for the final emotion classification task. This forces the descriptors to align logically with the actual emotional state.
[Input Audio] ➔ [Pre-trained SLM] ➔ [Confidence Filter] ➔ [RL SED Controller] ➔ [Rectified Labels] ➔ [Final SER-SLM SFT]
Datasets
The models were trained and evaluated using the following datasets:
- GigaSpeech-m (in SpeechCraft): A large-scale subset containing roughly 670k utterances, utilized for initial pre-training and training the Confidence Estimation Module (CEM).
- IEMOCAP: A widely used interactive emotional dyadic motion capture database used for post-training and evaluation.
- MELD: A multimodal multi-party dataset derived from the TV show Friends, used for post-training and evaluation.
Key Insights and Ideas
- The Threshold Paradox: When filtering data without the RL rectifier, retaining 80% of the data yielded the best results. However, when combined with the RL rectifier, the optimal retention rate shifted to 90%. This indicates that keeping slightly noisier data is beneficial if the model possesses an active, intelligent mechanism to fix labels on the fly.
- Tightening Emotion Boundaries: By forcing the acoustic descriptors to accurately reflect the emotional state via RL rewards, the model learns better latent representations. t-SNE visualizations showed that this method significantly sharpened the boundaries between easily confused emotion clusters, such as "Neutral" and "Happy."
Key Results
- Performance Gains: The integrated system achieved absolute accuracy improvements of 2.9% on IEMOCAP (reaching 80.98%) and 3.3% on MELD (reaching 64.11%) compared to baselines trained on full, unrectified datasets.
- SOTA Comparison: The proposed framework consistently outperformed several prominent open-source general-purpose and emotion-specific SLMs, including Qwen2-Audio, Kimi-Audio, and VIB-Emo.
- Policy Optimization: Ablation studies determined that sampling $M=6$ rectification policies per training step struck the ideal balance between computational efficiency and optimization accuracy.
Limitations and Future Work
- Computational Overhead: Generating multiple reinforcement learning policy samples ($M=6$) for every training step increases the computational footprint and time required for the post-training phase.
- Binary Policy Space: The current SED Controller only makes a binary choice per descriptor—either to retain the label or modify/flip it. This discrete approach may be too rigid for continuous acoustic features like highly variable pitch or volume contours.
- Future Directions: While not explicitly detailed, the methodology opens the door for extending on-the-fly rectification to continuous acoustic spaces and broader multi-modal datasets.
Summary updated