Improving recognition of speaker states and traits by cumulative evidence: intoxication, sleepiness, age and gender

Felix Weninger, Erik Marchi, Björn SchullerInterspeech 2012

TL;DR

This paper explores how to improve the accuracy of recognizing speaker states (intoxication, sleepiness) and traits (age, gender) by analyzing speech over longer periods. By fusing decisions from individual short speech segments, they demonstrate significant accuracy gains, especially for states like intoxication and sleepiness, suggesting a more robust and reliable approach to understanding speakers from their voice.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately identifying various speaker characteristics – specifically, temporary states like intoxication and sleepiness, and more permanent traits like age and gender – solely from speech. While short-term emotional states have been studied, medium-term states like intoxication and sleepiness, which can last for minutes to hours, and long-term traits, present a unique opportunity. The core problem is that decisions based on single, short speech utterances can be unreliable. The authors propose that by accumulating evidence over longer observation intervals, the accuracy of these classifications can be significantly improved, leading to more practical applications in areas like driver assistance, call center analysis, and forensics.

Methods

The authors employ a two-pronged approach:

  • Utterance-Level Classification: They first train classifiers on individual speech segments (utterances) to predict the speaker's state or trait. For intoxication and sleepiness, they use Support Vector Machines (SVMs) with a specific complexity constant, preserving the setup from previous challenges. For age and gender, they also use SVMs, but perform joint learning across all age and gender groups.
  • Cumulative Evidence Fusion: The key innovation is fusing the decisions from these utterance-level classifiers over time. For speaker states (intoxication, sleepiness), they use a majority vote across a series of utterances from a given session. For speaker traits (age, gender), they use a majority vote across utterances from a specific speaker. This cumulative approach aims to refine the initial, potentially noisy, utterance-level predictions.
  • Speaker Normalization: For speaker states (intoxication and sleepiness), they also investigate speaker normalization. This technique aims to reduce inter-speaker variability by normalizing feature values for each speaker, particularly when baseline speech from a "normal" condition is available. This is hypothesized to improve the separability of states like intoxication.

Datasets

The study utilizes three established corpora:

  • Alcohol Language Corpus (ALC): Used for evaluating intoxication recognition. This corpus contains speech from individuals at various blood alcohol concentrations (BAC). For the study, BAC was binarized into "alcoholized" and "non-alcoholized."
  • Sleepy Language Corpus (SLC): Used for evaluating sleepiness recognition. This corpus features speech from participants undergoing sleep deprivation, with sleepiness assessed using the Karolinska Sleepiness Scale (KSS) and binarized into "sleepy" and "non-sleepy."
  • Agender Database: Used for evaluating age and gender recognition. This corpus contains speech from individuals across different age groups (children, youth, adults, seniors) and genders.

The datasets were partitioned into training, development, and testing sets, with speaker-independent splits for the challenges.

Key Insights and Ideas

  • The Power of Time: The most significant insight is that analyzing speech over longer durations dramatically improves classification accuracy for speaker states and traits. This moves beyond snapshot analysis to a more holistic understanding of the speaker.
  • Cumulative Evidence is Key: Fusing decisions from multiple utterances (majority vote) is a highly effective strategy. It smooths out individual utterance errors and builds confidence in the prediction.
  • Speaker Normalization's Double-Edged Sword: Speaker normalization significantly boosts intoxication recognition by reducing speaker-specific voice characteristics. However, it surprisingly degrades sleepiness recognition, suggesting that the nature of these states and the corpora's characteristics (e.g., availability of a "sober" baseline for intoxication) play a crucial role.
  • States vs. Traits: While cumulative evidence benefits both, the improvements for medium-term states like intoxication and sleepiness are particularly pronounced, suggesting that their temporal nature makes them well-suited for this approach.
  • Efficiency Trade-off: The study highlights a trade-off between the time required for observation and the achieved accuracy. Even short observation periods (e.g., 30 seconds) yield substantial gains over single-utterance decisions.

Key Results

  • Intoxication: Cumulative evidence improved recall by up to 5% absolute. With speaker normalization and cumulative evidence, performance reached 78% UAR.
  • Sleepiness: Cumulative evidence led to significant improvements, reaching up to 75% UAR on the test set after about a minute of observation. Speaker normalization, however, decreased performance.
  • Age and Gender: Cumulative evidence provided substantial gains, with UAR jumping over 59% for age and 84% for gender after just 30 seconds of observation. Age recognition could be further improved to 61% UAR.
  • Efficiency: For age and gender, improvements were observed within the first 30 seconds of speech, suggesting that even short interactions can yield valuable insights.

Limitations and Future Work

  • Ground Truth Changes: The authors acknowledge that their current approach doesn't model ground truth changes within an observation interval. Future work could incorporate this, especially in interactive systems.
  • Optimal Unit of Analysis: Determining the ideal length of speech segments for utterance-level decisions to balance observation time and accuracy is an area for further research.
  • Context Modeling: The paper suggests that context modeling with recurrent neural networks (RNNs) could be a promising avenue for handling more complex scenarios, such as interactions with multiple speakers.
  • Real-life Application of Normalization: While speaker normalization shows promise for intoxication, its applicability in real-life scenarios where baseline speech might not be available needs further investigation.

Summary updated

BibTeX
@inproceedings{weninger2012improving,
  author = {Felix Weninger and Erik Marchi and Björn Schuller},
  title = {{Improving recognition of speaker states and traits by cumulative evidence: intoxication, sleepiness, age and gender}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure