Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources

Kate Forbes-Riley, Diane LitmanNAACL 2004

TL;DR

This paper explores how to automatically predict student emotions (positive, neutral, negative) in spoken tutoring dialogues. By combining acoustic-prosodic features (how things are said) with linguistic features (what is said) and contextual information, they achieve an impressive 84.75% accuracy, a significant improvement over previous methods. This work is crucial for developing more adaptive and effective intelligent tutoring systems.

What Problem Does This Paper Address?

Current intelligent tutoring systems (ITS) often fall short of human tutors, particularly in their ability to understand and respond to a student's emotional state. While ITS can deliver information and correct misconceptions, they typically lack the emotional intelligence that human tutors naturally employ. This paper aims to bridge this gap by developing a system that can automatically detect student emotions during spoken tutoring sessions. The motivation stems from the hypothesis that ITS could be significantly enhanced by modeling and adapting to student emotions, potentially leading to increased student persistence and learning. The challenge lies in accurately identifying these emotions from natural, spontaneous speech, which is far more complex than analyzing acted or read speech.

Methods

The researchers employed a machine learning approach, specifically using the AdaBoost algorithm with decision trees as the weak learner. They experimented with various combinations of features to predict emotion.

  • Feature Engineering: They extracted a rich set of features from student utterances:
  • Acoustic-Prosodic Features: These capture the "how" of speech, including pitch (fundamental frequency), energy (loudness), and temporal aspects like duration, speaking rate, and pausing. These features were normalized to account for individual speaker variations.
  • Non-Acoustic-Prosodic Features: These focus on the "what" of speech:
  • Lexical Features: Representing the words used in a turn as a word occurrence vector.
  • Automatic Features: Derived directly from transcriptions, such as turn begin/end times, and indicators of temporal overlap or barge-in.
  • Manual Features: Requiring human annotation initially, these include false starts, syntactic questions, semantic barge-ins, and canonical expressions (like "um" or "uh").
  • Contextual Features: To capture the flow of the dialogue, they incorporated features from the two preceding student turns (local context) and running averages/totals of features over all preceding turns (global context).
  • Identifier Features: Subject and problem identifiers were also included, as these can influence emotion expression.
  • Machine Learning: AdaBoost was chosen for its robust performance in combining weak learners to create a strong predictive model. They systematically evaluated different feature sets, from individual categories to comprehensive combinations.

Datasets

The study utilized a newly collected corpus of human-human spoken tutoring dialogues. This corpus consists of 149 dialogues from 17 university students who were new to physics. Each student turn was manually annotated for emotion (negative, neutral, positive) by two annotators. The final dataset for machine learning experiments comprised 385 student turns where both annotators agreed on the emotion label. This corpus serves as a benchmark for evaluating emotion prediction in a naturalistic tutoring context, differing from prior work that often used acted speech.

Key Insights and Ideas

  • The Power of "What" Over "How": While acoustic-prosodic features are useful, linguistic features (what is said) proved to be significantly more predictive of student emotion. This suggests that the content and structure of speech are stronger indicators than just the vocal characteristics.
  • Context is King (for Dialogue): Incorporating contextual information, particularly global features (running averages/totals), significantly boosted prediction accuracy, especially when combined with rich linguistic features. This highlights the importance of understanding the dialogue's history, not just the current utterance.
  • Combining Sources is Key: The best performance was achieved by combining acoustic-prosodic, text-based, and contextual features. This multi-modal approach leverages the strengths of each information source.
  • Identifier Features Matter (Sometimes): While identifier features (subject, problem) improved performance for some feature sets, their impact was most significant when combined with the most comprehensive feature sets, suggesting they provide crucial disambiguation in complex scenarios.
  • The "Alltext" Advantage: The "alltext" feature set, which included lexical, automatic, and manual linguistic features, consistently outperformed simpler text-based sets and even acoustic-prosodic features alone.

Key Results

  • The best performing model, using the "alltext+speech+glob-ident" feature set, achieved an accuracy of 84.75%.
  • This represents a 44% relative improvement in error reduction compared to a baseline classifier (which always predicted "neutral").
  • Text-based features alone consistently outperformed acoustic-prosodic features.
  • Adding contextual features (especially global) generally improved performance, with the "alltext+glob-ident" set being a strong performer.
  • Combining speech and text features showed marginal improvements over the best text-only features, but the difference was not always significant.

Limitations and Future Work

  • Manual Features: Some of the most effective linguistic features (e.g., false starts, semantic barge-ins) initially required manual annotation. Future work will focus on automatically deriving these features.
  • Emotion Annotation Schemes: The authors are exploring alternative emotion annotation schemes that capture more nuanced emotional dimensions or analyze emotions at smaller units than full turns.
  • Additional Features: They plan to incorporate a wider range of features, including language models, speaking style, dialog acts, part-of-speech tags, repetition, and emotionally salient keywords.
  • Data Fusion: Investigating methods for combining information beyond feature-level combination, such as fusing outputs from multiple classifiers, is a future direction.
  • Evaluation: The authors aim to compare feature set ordering based on recall, precision, and F-measure, not just accuracy.
  • Impact on Learning: A crucial next step is to investigate whether tutor responses to predicted emotions correlate with increased student learning.
  • Human-Computer Dialogues: The current study used human-human dialogues. Future work will extend these findings to human-computer dialogues generated by their ITSPOKE system.

Summary updated

BibTeX
@inproceedings{forbesriley2004predicting,
  author = {Kate Forbes-Riley and Diane Litman},
  title = {{Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure