TL;DR
This paper introduces a novel approach to multilingual emotion recognition in speech by leveraging biologically inspired "auditory attention" features. The authors demonstrate that these features can effectively capture emotional cues across languages, leading to robust monolingual, cross-lingual, and bilingual emotion recognition systems. This work contributes to building more universal and adaptable human-computer interaction systems.
What Problem Does This Paper Address?
Recognizing human emotions from speech is crucial for applications like personalized call centers, adaptive virtual agents, and dynamic video games. While significant progress has been made in monolingual emotion recognition, extending these systems to work across different languages (multilingual emotion recognition) remains a significant challenge. Existing methods often rely on traditional acoustic and prosodic features, which may not generalize well across linguistic boundaries. This paper aims to bridge this gap by exploring the effectiveness of auditory attention features for multilingual emotion recognition, hypothesizing that these biologically inspired features might capture more universal emotional cues.
Methods
The core of this work lies in the auditory attention (AA) model. This model is biologically inspired, mimicking early stages of human auditory processing. It begins by computing the auditory spectrum of the input sound using cochlear filtering. Then, it extracts multi-scale features (intensity, frequency contrast, temporal contrast, and orientation) from this spectrum using 2D spectro-temporal receptive filters, similar to those found in the primary auditory cortex.
These multi-scale features are then processed through a "center-surround" operation, which compares fine-scale features with coarser-scale features to detect local discontinuities. This process generates feature maps. From these maps, an "auditory gist" vector is extracted, representing the overall properties of each map. Finally, Principal Component Analysis (PCA) is used to reduce the dimensionality of these cumulative gist vectors, making them suitable for machine learning.
For emotion recognition, a three-layer neural network (3-NN) is employed. This network learns the mapping between the extracted auditory attention features and the emotion classes. The system operates at the syllable level, with the final sentence-level emotion tag determined by averaging posterior probabilities across all syllables in an utterance.
Datasets
The study utilizes two publicly available speech emotion datasets:
- Berlin Database of Emotional Speech (EMO-DB): This dataset contains German emotional speech from 10 actors, covering 7 emotions. For this study, 5 common emotions (neutral, anger, happiness, sadness, boredom) were used.
- LDC's Emotional Prosody Speech Corpus (Emo-Pro): This corpus contains English emotional speech from professional actors, with 15 emotion categories. Similar to EMO-DB, 5 common emotions (neutral, anger, happiness, sadness, boredom) were selected for the experiments.
The experiments were conducted using leave-one-speaker-out cross-validation to ensure speaker independence and provide a fair basis for cross-lingual comparisons.
Key Insights and Ideas
- Universality of Auditory Attention Features: The research suggests that auditory attention features, inspired by human auditory processing, are effective in capturing emotional cues that transcend language barriers. This is a key insight for developing more generalizable emotion recognition systems.
- Cross-Lingual Commonness: The experiments demonstrate that there is a degree of commonality in how emotions are expressed and detected in both German and English emotional speech, as evidenced by the performance of cross-lingual models.
- Bilingual Model Efficiency: A single bilingual model trained on data from both languages can achieve performance comparable to individual monolingual models, offering a more efficient solution by avoiding the need for separate models for each language.
- Auditory Attention vs. Traditional Features: The paper implicitly suggests that auditory attention features may outperform traditional prosodic and spectral features, especially in cross-lingual scenarios, by capturing richer, more universal emotional information.
Key Results
- Monolingual Performance: The system achieved high accuracies for monolingual emotion recognition: 82.7% on EMO-DB (German) and 56.7% on Emo-Pro (English) for five-class emotion classification.
- Cross-Lingual Performance: When tested across languages, accuracy dropped but remained significantly above chance: 55.1% for Emo-Pro (using German models) and 41.4% for EMO-DB (using English models).
- Bilingual Performance: The bilingual model achieved 78.6% accuracy on EMO-DB and 54.7% on Emo-Pro, demonstrating its ability to perform comparably to monolingual systems with a single model.
- Confusion Patterns: Anger was the most accurately detected emotion in both languages. Happiness was consistently the most confused emotion, often with anger, in both monolingual and bilingual settings.
Limitations and Future Work
- Dataset Limitations: The study acknowledges that the Emo-Pro dataset contains simulated emotional speech, which might differ from spontaneous emotional speech.
- Future Directions: The authors propose several avenues for future research:
- Including data from more languages, particularly tonal languages like Mandarin, to investigate the impact of tone on multilingual emotion recognition.
- Integrating facial features with acoustic features to create a unified, multimodal system that performs well across languages and emotions.
- Exploring smaller bilingual models that maintain performance while reducing computational complexity.
Summary updated