Emotion recognition using acoustic and lexical features

Viktor Rozgić, Sankaranarayanan Ananthakrishnan, Shirin Saleem, Rohit Kumar, Aravind Namandi Vembu, Rohit PrasadInterspeech 2012

TL;DR

This paper presents a novel approach to emotion recognition in speech by fusing acoustic features with lexical information extracted from Automatic Speech Recognition (ASR) output. By combining acoustic cues with word-based sentiment and presence indicators, they achieve state-of-the-art results on a challenging emotion recognition dataset, demonstrating the power of multi-modal feature fusion.

What Problem Does This Paper Address?

Recognizing human emotions from speech is a complex task with applications ranging from virtual training to mental health diagnostics. While both what is said (linguistic content) and how it's said (acoustic properties) carry emotional information, effectively combining these cues has been a challenge. Previous work often relied on perfectly transcribed speech, which is unrealistic in many real-world scenarios. This paper tackles the problem of robust, utterance-level emotion recognition by integrating acoustic features with lexical information derived from imperfect ASR outputs, aiming to improve accuracy and overcome the limitations of relying solely on acoustic cues.

Methods

The authors propose a multi-faceted feature extraction and fusion strategy:

  • Acoustic Features:
    • Low-Level Descriptors (LLD): Standard acoustic features like energy, pitch (F0), formants, and Mel-Frequency Cepstral Coefficients (MFCCs) are extracted. Statistical functionals (mean, standard deviation, etc.) are computed over these frame-level features to create utterance-level representations. Voice quality features like jitter and shimmer, known to correlate with negative emotions, are also included.
  • Model-Based MFCC Features (M-MFCC): To overcome the potential oversimplification of statistical functionals for non-stationary features like MFCCs, they introduce a novel approach. They train emotion-dependent Gaussian Mixture Models (GMMs) for each emotion category. Then, frame-level MFCCs are scored against these models, generating likelihood vectors. These likelihoods are then fused into utterance-level features using three methods: mean of normalized likelihoods, histograms of highest-scoring models, and Dirichlet distribution parameter estimation. This method aims to capture more nuanced emotional information embedded within the MFCCs.
  • Lexical Features:
    • ASR-Derived Features (LEX-ASR): Lexical information is extracted from the output of an Automatic Speech Recognition (ASR) system. This includes binary indicators for the presence or absence of stemmed word stems and sentiment categories derived from bag-of-words representations using lexicons like LIWC and General Inquirer.
  • Reference Transcript Features (LEX-T): For comparison, similar lexical features are also extracted from manually provided reference transcripts.
  • Fusion and Classification:
    • The acoustic (ACO), model-based acoustic (M-MFCC), and lexical (LEX-ASR or LEX-T) features are concatenated to form a comprehensive feature vector.
  • Support Vector Machines (SVMs) with a Radial Basis Function (RBF) kernel are used for emotion classification. Hyperparameter optimization (C and gamma) is performed using cross-validation.

Datasets

The primary dataset used for evaluation is the USC-IEMOCAP database. This database contains approximately 12 hours of dyadic interactions between actors, featuring both scripted and improvised dialogues. Utterances are manually transcribed and annotated with categorical emotion labels (anger, happiness, sadness, excitement, neutral) and dimensional attributes (valence, activation, dominance). For this study, the happiness and excitement categories were merged, resulting in a four-way classification task (anger, happiness, sadness, neutral).

Key Insights and Ideas

  • The Power of Fusion: The paper strongly advocates for combining acoustic and lexical features. While acoustic features alone are informative, integrating lexical cues, especially those derived from ASR, significantly boosts performance.
  • Model-Based MFCCs Offer an Edge: The novel model-based MFCC features (M-MFCC) outperform traditional statistical functionals of MFCCs, suggesting that directly modeling emotion-specific patterns within MFCCs is more effective than simply summarizing their statistical properties.
  • ASR Output is Viable: Despite a high ASR word error rate (40%) on this spontaneous speech dataset, the ASR-derived lexical features still contribute positively to emotion recognition, demonstrating the robustness of this approach even with imperfect transcriptions.
  • Complementary Strengths: Lexical features, particularly when derived from reference transcripts, show particular strength in recognizing happiness and neutral emotions, which were previously the weakest categories for acoustic features alone. This highlights how different modalities capture different aspects of emotional expression.

Key Results

The paper reports state-of-the-art results on the USC-IEMOCAP dataset for four-way emotion recognition:

  • Acoustic Features (ACO + M-MFCC): Achieved an unweighted accuracy of 60.2%, outperforming previous acoustic-only results on this dataset.
  • Fusion with ASR Lexical Features (ACO + M-MFCC + LEX-ASR): This combination yielded an unweighted accuracy of 65.7%, which is the main reported result and the paper's key contribution.
  • Fusion with Reference Transcript Lexical Features (ACO + M-MFCC + LEX-T): Achieved an even higher unweighted accuracy of 70.1%.

The results also show that lexical features, especially transcript-based ones, significantly improve recognition for happiness and neutral categories.

Limitations and Future Work

  • ASR Performance: The authors acknowledge the high word error rate of the ASR system on the spontaneous speech data, which likely limits the full potential of the ASR-derived lexical features.
  • Dataset Specificity: While USC-IEMOCAP is a well-regarded dataset, the findings might not generalize directly to all types of emotional speech (e.g., acted vs. spontaneous, different languages).

Future work directions include:

  • Exploring different models for generating model-based features.
  • Integrating visual features from motion capture data.
  • Identifying specific frame subsets that are most crucial for utterance-level emotion recognition.

Summary updated

BibTeX
@inproceedings{rozgic2012emotion,
  author = {Viktor Rozgić and Sankaranarayanan Ananthakrishnan and Shirin Saleem and Rohit Kumar and Aravind Namandi Vembu and Rohit Prasad},
  title = {{Emotion recognition using acoustic and lexical features}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure