Incremental acoustic valence recognition: an inter-corpus perspective on features, matching, and performance in a gating paradigm

Björn Schuller, Laurence DevillersInterspeech 2010

TL;DR

This paper investigates how quickly emotion recognition systems can reliably detect valence (positive/negative) in speech, finding that about one second of speech is sufficient. They also explore whether matching the training data's length to the test data's length improves performance, with mixed results depending on the data source and feature set.

What Problem Does This Paper Address?

Many real-time applications, like virtual agents, need to understand a user's emotional state very quickly, often before they even finish speaking. This is crucial for providing timely and appropriate responses. However, most emotion recognition research focuses on analyzing complete utterances. This paper tackles the challenge of "incremental" emotion recognition, specifically focusing on valence, by examining how performance changes as more speech becomes available over time. This is akin to a "gating paradigm" where the system's understanding evolves with incoming data. The authors also investigate whether training models on speech segments of similar lengths to those being tested improves accuracy, a concept known as "matched length training."

Methods

The core of the methodology involves an "incremental" or "gating" approach. This means the system is evaluated on speech segments of increasing lengths, starting from very short durations (e.g., 0.1 seconds) up to the full utterance. Acoustic features are extracted from these segments, and the performance of emotion recognition models (specifically Support Vector Machines) is measured at each time step.

Key aspects of their method include:

  • Feature Analysis: They analyze various groups of acoustic features (low-level descriptors like pitch, loudness, MFCCs, and functionals derived from them) to see which ones contribute most to early emotion detection and how their performance evolves over time.
  • Matched Length Training: They compare two training strategies:
  • Standard Training: Models are trained on full-length utterances.
  • Matched Length Training: Models are trained on segments whose lengths are matched to the test segment lengths. This requires a reliable speech onset detection mechanism.
  • Cross-Corpus Evaluation: They use two distinct French speech corpora (CINEMO and JEMO) to ensure their findings are generalizable and to study the impact of different data characteristics (manual vs. automatic segmentation) and inter-corpus performance.

Datasets

The study utilizes two French speech corpora:

  • CINEMO: This corpus contains manually segmented speech instances. It features 3,992 instances from 51 speakers, with a total net playtime of over 2 hours. The emotions considered for valence are ANGER, NEUTRAL, and HAPPINESS.
  • JEMO: This corpus consists of automatically segmented speech turns from 39 speakers. It has 1,134 instances totaling about 38 minutes of playtime. Similar to CINEMO, it focuses on ANGER, NEUTRAL, and JOY for valence analysis. The JEMO corpus was collected in an emotion-detection game where players acted emotions, leading to potentially more expressive and spontaneous utterances.

Key Insights and Ideas

  • One Second is Enough (Mostly): For valence recognition, approximately one second of speech is sufficient to achieve a performance level close to that obtained with the full utterance. Beyond this point, the performance gains are marginal.
  • Feature Group Behavior Varies: Different acoustic feature groups exhibit distinct "gradient behaviors." For instance, pitch-related features show a flatter performance curve as more speech is available, suggesting they are informative early on. MFCCs, on the other hand, show a steeper initial rise before saturating.
  • Matched Length Training is Context-Dependent: The benefit of matching training segment lengths to test segment lengths is not universal. It proved beneficial for the CINEMO corpus (manually segmented) but was counter-productive for the JEMO corpus (automatically segmented). This suggests that the quality and nature of segmentation play a significant role.
  • Inter-Corpus Performance is Challenging: Recognizing emotions across different corpora (e.g., training on CINEMO and testing on JEMO) results in significantly lower performance, highlighting the impact of data diversity and segmentation differences.

Key Results

  • Sufficient Duration: The study demonstrates that about 1 second of speech is sufficient for reliable valence recognition, with performance saturating thereafter.
  • Feature Ranking: Pitch and its envelope features are found to be among the most stable and informative for early emotion detection.
  • Matched Length Gains: For CINEMO, matched length training showed performance gains, particularly at the beginning of utterances. However, for JEMO, this approach led to a decrease in performance.
  • WAR/UAR Performance: Table 3 shows that for merged corpora (training and testing on both), the Weighted Average Recall (WAR) reached around 60.4% to 77.3% depending on the specific train/test split and length. Unweighted Average Recall (UAR) was generally lower, reflecting class imbalance.

Limitations and Future Work

  • Limitations:
    • The study primarily focuses on valence, and findings might differ for other emotional dimensions.
  • The "matched length training" approach relies on accurate speech onset detection, which is not explicitly addressed in this paper.
  • The performance gains from matched length training were inconsistent across corpora, suggesting further investigation into why.
  • Future Work:
    • Investigate other segmentations, such as voiced/unvoiced segments or fixed-length intervals, rather than just time-based gating.
  • Explore the gating paradigm for other emotion categorization tasks beyond valence.
  • Further analyze the interaction between feature groups and segmentation types.

Summary updated

BibTeX
@inproceedings{schuller2010incremental,
  author = {Björn Schuller and Laurence Devillers},
  title = {{Incremental acoustic valence recognition: an inter-corpus perspective on features, matching, and performance in a gating paradigm}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure