Domain adaptation and compensation for emotion detection

Michelle Hewlett Sanchez, Gökhan Tür, Luciana Ferrer, Dilek Hakkani-TürInterspeech 2010 · OpenReview

Abstract

Emotion detection in high-stakes, real-world environments like 911 emergency calls is often hampered by the scarcity of labeled spontaneous speech data. This study explores leveraging larger, "out-of-domain" acted speech datasets to improve classification performance in target domains. By employing domain adaptation and Nuisance Attribute Projection (NAP) to filter out dataset-specific variations, the researchers significantly enhanced the detection of fear in emergency callers, achieving a peak F-measure of 74.5% through call-level post-processing.

Introduction

Most emotion detection research relies on "acted" speech recorded in controlled laboratory settings. While these datasets are extensive, they often fail to capture the acoustic and emotional complexity of spontaneous human emotion. Real-world data, such as 911 emergency calls, is noisy and difficult to annotate, leading to a significant "domain mismatch" when models trained on lab data are applied to the field. This research addresses how to effectively use clean, out-of-domain datasets to improve models for small, noisy, in-domain datasets, specifically focusing on distinguishing fear from neutral states in emergency callers.

Methods

The study utilized a Support Vector Machine (SVM) classifier with prosodic features including pitch (F0), energy (RMS), and speaking rate. Two primary strategies were tested to bridge the domain gap:

  1. Domain Adaptation: This included simple data concatenation and a "Decision-as-Feature" ($LDC \rightarrow 911$) approach, where predictions from an out-of-domain model served as inputs for the in-domain model.
  2. Domain Compensation (NAP): Nuisance Attribute Projection (NAP) was adapted from speaker recognition. In this novel application, the domain difference itself was treated as a "nuisance" to be projected out of the feature space, allowing the model to focus on underlying emotional markers rather than recording conditions.

A final post-processing step aggregated segment-level predictions to the call level using majority voting to improve overall robustness.

Datasets

The researchers utilized two primary corpora: the in-domain 911 Corpus and the out-of-domain LDC Emotional Prosody Corpus. The 911 data is characterized by its spontaneous nature and telephone-grade audio, while the LDC data consists of balanced, acted speech.

Distribution of segments and calls across the 911 and LDC datasets, categorized by gender and emotion.

Due to the rarity of certain emotions in the 911 corpus, the study focused on the binary classification of "fear/panic" versus "neutral" states.

Results and Discussion

Performance was evaluated using the F-measure for the "fear" class. The results demonstrate that both domain adaptation and compensation techniques outperform the baseline and models trained solely on in-domain data.

F-measure results for fear detection across various experimental setups, showing the impact of domain adaptation and post-processing.

Key findings include:

  • NAP Effectiveness: Treating the domain as a nuisance attribute allowed the model to ignore recording quality and acting styles, focusing on prosodic markers of fear. The NAP 911+LDC approach reached a 64.8% F-measure at the segment level.
  • Decision-as-Feature: The $LDC \rightarrow 911$ approach proved highly effective, suggesting that even imperfect lab-trained models provide valuable "hints" that a second model can calibrate for real-world use.
  • Post-processing Impact: Aggregating segment predictions to the call level provided the most significant performance boost, with the $LDC \rightarrow 911$ method reaching a peak F-measure of 74.5%.

Conclusion

This work highlights the potential of domain compensation techniques like NAP to bridge the gap between laboratory and real-world emotional data. While the results are based on a relatively small sample size, they suggest that prosodic features, when properly adjusted for domain bias, are powerful indicators of high-arousal states like fear. Future work will explore the integration of lexical and contextual features to further refine detection in complex emergency scenarios.

Summary updated

BibTeX
@inproceedings{sanchez2010domain,
  author = {Michelle Hewlett Sanchez and Gökhan Tür and Luciana Ferrer and Dilek Hakkani-Tür},
  title = {{Domain adaptation and compensation for emotion detection}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=NgiJXOOr62R},
}
Figure