Detecting Emotion in Speech: Experiments in Three Domains

Jackson LiscombeNAACL 2006

TL;DR

This paper explores how to detect emotions in spoken language by extracting acoustic and linguistic features across three different datasets. The authors demonstrate that combining various features improves emotion classification accuracy and propose that integrating emotion detection into spoken dialogue systems, like intelligent tutors, can enhance user experience and learning outcomes.

What Problem Does This Paper Address?

Human speech is rich with emotional cues, yet current spoken dialogue systems often fail to recognize or respond to these emotions, leading to unnatural and sometimes inappropriate interactions. This disconnect hinders the development of truly engaging and effective human-computer communication. The paper tackles two fundamental questions: 1) How is emotion conveyed in speech, and 2) Can modeling emotion improve spoken dialogue applications? The authors aim to identify key features that characterize emotional speech and to empirically demonstrate the benefits of incorporating emotion awareness into intelligent tutoring systems.

Methods

The research employs a standard machine learning paradigm: feature extraction followed by classification. For emotion classification, the authors experiment with various feature sets:

  • Global Acoustic-Prosodic Features: These are well-established features like speaking rate, pitch (minimum, maximum, mean), and intensity.
  • Linguistic Features: This includes TOBI labels (a system for transcribing prosody) and spectral tilt of vowels.
  • Transcriptive Features: Lexical items, filled pauses, and non-speech noises are considered.
  • Dialogue Act Features: The communicative function of an utterance within a dialogue.
  • Contextual Features: Tracking the history of acoustic-prosodic and transcriptive features over previous turns.
  • Breath-Group Features: A novel approach to segmenting utterances into intonationally meaningful units by identifying pauses, which are then used to extract features like pause time and global prosodic information within these segments.
  • Pitch Contour Clustering: A proposed method to group similar pitch contours based on abstract form using edit distance and dynamic time warping.
  • Voice Quality Features: Investigating features derived from the glottal pulse waveform, approximated through inverse filtering of the speech signal.

Machine learning algorithms like RIPPER (a rule-learning algorithm) and BOOSTEXTER (an iterative hypothesis-forming algorithm) are used for classification.

Datasets

The research utilizes three distinct datasets:

  • EPSaT (Emotional Prosody Speech and Transcription corpus): Recordings of professional actors reading short utterances with different emotional states. This corpus was used for initial feature exploration and subjective labeling of 10 emotions.
  • HMIHY (How May I Help You?): A corpus of natural human-computer dialogues from AT&T customers interacting with an automated agent. This dataset provided spontaneous speech with annotations for seven emotional labels, which were later collapsed into negative and non-negative categories due to label skew.
  • ITSpoke (Intelligent Tutoring Spoken Dialogue System): Data from an intelligent tutoring system, specifically human-human tutorial dialogues. This corpus was annotated for anger, frustration, and uncertainty, and served as the platform for implementing and evaluating emotion-aware responses.

Key Insights and Ideas

  • Multifaceted Nature of Emotional Speech: No single feature set is sufficient for robust emotion detection. Combining acoustic, prosodic, linguistic, and contextual information significantly improves classification accuracy.
  • Domain-Specific vs. Universal Features: While some acoustic-prosodic features show consistent predictive power across domains, the effectiveness of other features (like transcriptive or contextual) may be more domain-dependent. The research aims to identify universal emotional cues.
  • The Promise of Breath-Group Features: The introduction of breath-group segmentation and its associated features shows potential for improving emotion classification, suggesting that intonational units are important carriers of emotional information.
  • Emotion Modeling in Tutoring Systems: The hypothesis that detecting and responding to student emotions (like frustration and uncertainty) can positively impact learning outcomes and student engagement in intelligent tutoring systems is a key driver for the implementation work.

Key Results

  • EPSaT: Achieved an average of 75% prediction accuracy for individual emotions, representing a 22% improvement over chance. Negative emotions correlated with plateau boundary tones, while positive emotions correlated with standard declarative phrasal endings.
  • HMIHY: Classification accuracy reached 79% when combining acoustic-prosodic, lexical, pragmatic, and contextual features, a 5% improvement over acoustic-prosodic features alone and an 8% improvement over the majority class baseline.
  • ITSpoke: Automatic acoustic-prosodic features achieved 75% accuracy, a 13% improvement over the baseline. Adding contextual and breath-group features improved accuracy to 77%.

Limitations and Future Work

  • Subjectivity of Labels: The reliance on subjective human annotations for emotion labels can introduce variability and bias.
  • Domain Specificity: While the study explores three domains, a more comprehensive analysis across a wider range of domains is needed to establish universal emotion communication patterns.
  • Implementation Constraints: In the ITSpoke implementation, the authors are limited to inserting "active listening prompts" rather than altering the core discourse flow, which might limit the full impact of emotion-aware responses.
  • Future Work: The authors plan to further explore novel features like pitch contour clustering and voice quality estimation. They also aim to implement and evaluate an emotion detection and response module within ITSpoke, measuring learning gains and other behavioral metrics to provide empirical evidence for the benefits of emotion modeling in spoken dialogue systems.

Summary updated

BibTeX
@inproceedings{liscombe2006detecting,
  author = {Jackson Liscombe},
  title = {{Detecting Emotion in Speech: Experiments in Three Domains}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure