Analyzing dialogue data for real-world emotional speech classification

Ryuichi Nisimura, Souji Omae, Hideki Kawahara, Toshio IrinoInterspeech 2006

TL;DR

This paper tackles the challenge of classifying emotions in real-world spoken dialogue, moving beyond lab-acted speech. They introduce "delightful" and "hateable" as key psychological factors derived from Japanese child voices, achieving a remarkable 98.8% accuracy in distinguishing these emotions using 11 acoustic features and an SVM classifier.

What Problem Does This Paper Address?

Developing intelligent human-machine dialogue systems requires understanding the user's emotional state. While previous research often relied on acted emotions, this paper addresses the gap in analyzing real-world emotional speech, specifically within the context of a public spoken dialogue system. The challenge lies in collecting and interpreting genuine emotional expressions, which are more nuanced and less exaggerated than acted ones. The authors aim to bridge the understanding of subjective emotional experiences with objective acoustic features to enable more empathetic and responsive dialogue systems.

Methods

The core of the methodology involves a three-step process:

  1. Emotional Evaluation: 16 basic emotions were evaluated on a 5-grade scale by an operator for Japanese child voices collected from a real-world dialogue system. This subjective evaluation forms the basis for understanding emotional states.
  2. Factor Analysis: A factor analysis (specifically, varimax rotation with maximum likelihood method and Bartlett scoring) was employed to reduce the 16 emotional dimensions into fundamental psychological factors. This is a standard statistical technique for identifying underlying latent variables.
  3. Acoustic Feature Extraction and Classification: After identifying key factors, the researchers extracted 16 acoustic features (including pitch, pitch rising slopes, power, and power rising slopes, along with their maximum, median, interquartile range, and variance). A Support Vector Machine (SVM) with an ANOVA kernel was then used as the classifier to discriminate between the derived emotional factors. The SVM is a well-established machine learning algorithm for classification tasks.

Datasets

The primary dataset used in this study consists of natural dialogue utterances collected from the "Takemaru-kun" public spoken dialogue system. Specifically, 2,699 sentences uttered by children between April 1 and April 20, 2003, were used for evaluation. The authors highlight the importance of using child voices because they are less likely to feign emotions compared to adults, making their expressions more genuine. This dataset represents a novel benchmark for real-world emotional speech analysis in children.

Key Insights and Ideas

  • "Delightful" vs. "Hateable" as Core Emotional Dimensions: The factor analysis revealed that the complex landscape of 16 basic emotions could be effectively summarized by two primary psychological factors: "delightful" (associated with positive emotions like pleasure, mirth, and joy) and "hateable" (associated with negative emotions like anger, pressure, and contempt). This simplification offers a more manageable framework for emotion recognition.
  • Child Voices as a Rich Source of Genuine Emotion: The study emphasizes the value of using children's natural utterances for emotion analysis, as they are less prone to conscious emotional masking, providing a more authentic representation of real-world emotional states.
  • Predictive Power of Specific Acoustic Features: The research identified a surprisingly small set of 11 acoustic features that are highly predictive of these emotional states, particularly the variances of power and power rising slopes, which significantly improved classification accuracy.

Key Results

The study achieved a high correct rate of 98.8% in classifying child utterances into either "delightful" or "hateable" emotions. This was accomplished using a Support Vector Machine (SVM) classifier with only 11 carefully selected acoustic features. The classification was performed on a dataset of 600 samples (300 delightful, 300 hateable), averaged using a jackknife method.

Limitations and Future Work

Limitations:

  • The study acknowledges that the factor analysis results for "Neutral" showed independent tendencies, suggesting further investigation is needed.
  • The current analysis is primarily based on child utterances; the authors note that adult utterances would require further investigation.
  • The contribution rates of the factors were noted as being low, suggesting that improvements to the analysis method could further enhance results.

Future Work:

  • Conduct further analysis considering other factors identified in Section 3.
  • Investigate adult utterances to understand emotional expression differences.
  • Evaluate the proposed emotional classification scheme within the real-world spoken dialogue system.

Summary updated

BibTeX
@inproceedings{nisimura2006analyzing,
  author = {Ryuichi Nisimura and Souji Omae and Hideki Kawahara and Toshio Irino},
  title = {{Analyzing dialogue data for real-world emotional speech classification}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure