Prosodic cues to disengagement and uncertainty in physics tutorial dialogues

Diane Litman, Heather Friedberg, Kate Forbes-RileyInterspeech 2012

TL;DR

This paper investigates how the sound of a student's voice (prosody) can reveal their disengagement and uncertainty during physics tutoring sessions. By analyzing acoustic features like pitch and duration, the researchers found that these vocal cues can be used to predict these crucial affective states, potentially allowing AI tutors to adapt their teaching strategies for better learning outcomes.

What Problem Does This Paper Address?

Traditional AI tutors often focus solely on whether a student's answer is correct. However, a student's emotional and cognitive state – like confusion or boredom – significantly impacts their learning. While previous research has explored detecting emotions in customer service or general conversation, there's a need to understand these states specifically within educational contexts, particularly in spoken dialogue systems. This paper tackles the challenge of detecting "disengagement" (lack of effort or interest) and "uncertainty" (confusion about the material) in students interacting with a physics tutor, using only their speech. The goal is to see if vocal cues can reliably signal these states, which are known to negatively affect learning.

Methods

The researchers analyzed a corpus of spoken physics tutorial dialogues. They used a trained annotator to label student turns as either "disengaged" (DISE) or "engaged" (ENG), and "uncertain" (UNC) or "certain" (CER).

The core of their method involved extracting a set of acoustic-prosodic features from the student's speech. These features fall into three categories:

  • Temporal: Turn duration and the duration of the pause before a student's turn.
  • Pitch (f0): Measures like maximum, minimum, mean, and standard deviation of the fundamental frequency.
  • Energy (RMS): Measures like maximum, minimum, mean, and standard deviation of the speech signal's loudness.

They then used these features in two ways:

  1. Descriptive Analysis: They compared the average prosodic features for disengaged vs. engaged turns, and uncertain vs. certain turns, using statistical tests to identify significant differences.
  2. Predictive Modeling: They trained decision tree models to predict disengagement and uncertainty using these prosodic features. To account for speaker variability, they normalized feature values by the student's initial turn values. They also used a cost matrix to penalize misclassifications of the less frequent states (DISE and UNC).

Datasets

The study used a newly collected corpus of spoken dialogues from the ITSPOKE (Intelligent Tutoring SPOKEn dialogue system). This corpus consists of:

  • 432 dialogues
  • 7216 student turns
  • From 72 students (47 female, 25 male)
  • Students were college-level with no prior physics knowledge.

The dialogues were collected during an experiment evaluating the ITSPOKE system's ability to detect and adapt to student uncertainty. The corpus is annotated with binary labels for disengagement (DISE/ENG) and uncertainty (UNC/CER).

Key Insights and Ideas

  • Vocal Cues Matter in Tutoring: The study strongly suggests that the way a student speaks, not just what they say, provides valuable information about their learning state.
  • Disengagement and Uncertainty Have Distinct (and Overlapping) Prosodic Signatures: While both states are associated with longer response times and lower pitch, uncertainty also correlates with softer speech, whereas disengagement doesn't show a clear difference in loudness compared to engaged speech.
  • Temporal Cues are King (for Disengagement): Response time (turn duration and prior pause) appears to be the most crucial feature for predicting disengagement, playing a central role in the predictive models.
  • Pitch is Key for Uncertainty: Pitch-related features are more prominent in predicting student uncertainty.
  • Context is Crucial for Prediction: While individual prosodic features show differences, combining them in predictive models yields better results. The best combination of features differs between predicting disengagement and uncertainty.

Key Results

  • Descriptive Analysis:
    • Disengaged (DISE) vs. Engaged (ENG): Disengaged turns had significantly longer prior pauses, lower mean and max pitch, and less pitch and energy variation.
  • Uncertain (UNC) vs. Certain (CER): Uncertain turns also had significantly longer prior pauses, lower mean and max pitch, and less pitch and energy variation. Additionally, uncertain turns were spoken with lower maximum and mean energy (softer).
  • Predictive Modeling:
    • Decision tree models using acoustic-prosodic features achieved unweighted average precision and recall of 63% and 61% for uncertainty, and 61% and 56% for disengagement.
  • These results significantly improve over majority class baselines.
  • Temporal features were the most queried in both models, but played a more central role in the disengagement model. Pitch features were more prominent in the uncertainty model.

Limitations and Future Work

  • Corpus Size: The authors acknowledge that the corpus size is relatively small, which may have limited the performance gains from adding more complex features.
  • Feature Combination: While acoustic-prosodic features show utility, their absolute performance can be increased by incorporating lexical, semantic, and discourse features.
  • Generalizability: The authors plan to replicate these analyses on other corpora to see if these prosodic findings generalize across different dialogue systems and domains.
  • Real-time Implementation: They are currently implementing their best predictive models into the ITSPOKE system to evaluate the practical utility of detecting and adapting to uncertainty and disengagement in a controlled experiment.

Summary updated

BibTeX
@inproceedings{litman2012prosodic,
  author = {Diane Litman and Heather Friedberg and Kate Forbes-Riley},
  title = {{Prosodic cues to disengagement and uncertainty in physics tutorial dialogues}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure