Introduction
Automated emotion recognition typically leverages two distinct modalities: acoustics (prosody, pitch, energy) and linguistics (lexical content). While acoustic features are readily extracted from audio signals, linguistic analysis necessitates a text transcript. Manual transcription is prohibitively expensive and slow for large-scale or real-time applications, positioning Automatic Speech Recognition (ASR) as a vital component of the pipeline. However, ASR systems are prone to errors, particularly in spontaneous speech or when processing children's voices. This paper investigates the robustness of linguistic emotion recognition models when faced with imperfect ASR output, identifying the performance thresholds and strategies to mitigate error propagation.
Methodology
The study evaluates two primary linguistic approaches to emotion classification:
- Emotional Salience Classifier: A keyword-based approach that identifies words or bi-grams statistically associated with specific emotional states.
- Bag of Words (BOW) Classifier: A vector-space model representing word occurrences, paired with back-end classifiers such as Support Vector Machines (SVM) and Discriminative Multinomial Naive Bayes (DMNB).
To simulate varying levels of ASR quality, the researchers utilized two different ASR engines and adjusted decoding parameters to produce transcripts with a wide range of Word Accuracy (WA).
Experimental Setup: The FAU Aibo Emotion Corpus
The models were tested on the FAU Aibo Emotion Corpus, which contains recordings of 51 German children interacting with a Sony Aibo robot. The children were led to believe the robot was autonomous, while it was actually controlled via a "Wizard-of-Oz" setup designed to elicit emotional responses through disobedient behavior. The original 11 emotional categories were mapped to a binary classification task: NEGative (comprising angry, touchy, reprimanding, and emphatic states) versus IDLe (neutral and other non-negative states).
Results and Discussion
Performance Thresholds and Robustness
A key finding is the remarkable resilience of linguistic cues to ASR degradation. Emotion recognition performance, measured by Unweighted Average Recall (UAR), remains relatively stable as long as the ASR Word Accuracy stays above a certain threshold. For the Emotional Salience model, this threshold is approximately 65% WA, while the BOW-SVM model requires roughly 77% WA to maintain baseline performance levels.
The "Matched Training" Advantage
The researchers observed that training emotion models on ASR-generated transcripts ("matched training") often yields superior results compared to models trained on perfect manual transcripts ("best training"), provided the test data is also ASR-generated. This suggests that the classifier learns to accommodate the systematic biases and errors inherent to the specific ASR engine. At lower word accuracies, the "matched" models significantly outperform those trained on clean data.
Systematic Errors as Discriminative Features
In some cases, ASR errors actually serve as reliable indicators of emotion. For instance, the German word "nein" (no) was frequently misrecognized as "Neid" (grudge) when spoken with negative affect. Because this specific misrecognition occurred consistently in negative contexts, the classifier treated "Neid" as a strong feature for the NEGative class, effectively turning a recognition failure into a classification asset.
Conclusion
Linguistic-based emotion recognition is surprisingly robust to ASR errors. By employing "matched training" strategies, developers can build systems that maintain high accuracy even with imperfect transcripts. Future work will explore multi-class emotion detection and the integration of ASR confidence scores to further enhance the fusion of acoustic and linguistic features.
Summary updated