Active learning by label uncertainty for acoustic emotion recognition

Zixing Zhang, Jun Deng, Erik Marchi, Björn SchullerInterspeech 2013

TL;DR

This paper introduces a novel active learning approach for acoustic emotion recognition that leverages "label uncertainty" – the inherent ambiguity in human judgments of emotion. By modeling this uncertainty, their method intelligently selects the most informative speech segments for human labeling, significantly reducing annotation costs while improving classifier performance compared to random selection.

What Problem Does This Paper Address?

Training accurate acoustic emotion recognition systems requires large amounts of labeled speech data. However, collecting and annotating this data is a time-consuming and expensive bottleneck, often consuming the majority of a project's effort. Furthermore, emotional labels are inherently subjective, leading to disagreements among human annotators. This "label uncertainty" is a significant challenge. Traditional active learning methods, which typically rely on classifier confidence or agreement among multiple classifiers, are often applied to objective tasks. This paper tackles the problem of efficiently building robust emotion classifiers by specifically addressing the subjective nature of emotional labels and the resulting label uncertainty.

Methods

The core of this paper is a novel active learning (AL) strategy that exploits label uncertainty. Instead of relying on a single classifier's confidence, their approach models the agreement levels among multiple human annotators.

Here's a breakdown of their method:

  • Uncertainty Modeling: They build "uncertainty models" based on the labeller agreement levels for instances in the initial labeled training set. These models aim to predict how likely an unlabelled instance is to fall into different emotional categories based on the historical agreement patterns.
  • Instance Selection: For unlabelled data, they use these uncertainty models to predict labeller agreement levels. Based on these predictions, they employ two selection strategies:
  • Class Sparseness (ALcs): This method prioritizes selecting instances that are predicted to belong to the "sparse" class (i.e., the less frequent class, like "negative" emotions in their dataset). The idea is to actively seek out examples that are harder to classify or less represented.
  • Confidence Values (ALcv): This method selects instances whose predicted labeller agreement levels fall within a specific range, aiming to avoid instances with very low confidence (which might be noisy) and very high confidence (which might be redundant).
  • Iterative Process: The selected instances are then sent for human labeling, added to the training set, and the process repeats until a stopping criterion (e.g., performance target, maximum iterations) is met.

Datasets

  • FAU Aibo Emotion Corpus (AEC): This is the primary dataset used for training and evaluation. It consists of spontaneous child-child interactions with a Sony Aibo robot, recorded in German. The corpus contains 18,216 chunks labeled with emotional states. For their experiments, they used a 2-class labeling: NEGative (angry, touchy, reprimanding, emphatic) and IDLe (all other states).
  • Data Partitioning: They split the AEC into a "pool" (used for candidate instances) and a "test set." To ensure speaker independence, the pool data came from one school (OHM) and the test data from another (MONT).

Key Insights and Ideas

  • Leveraging Subjectivity: The paper demonstrates that the inherent subjectivity and disagreement in human emotion labeling (label uncertainty) can be a valuable signal for active learning, rather than just a problem to be overcome.
  • Class Sparseness is Key: The "class sparseness" selection method (ALcs) proved particularly effective, suggesting that actively seeking out less common or more ambiguous emotional expressions is crucial for building robust classifiers.
  • Efficiency Gains: Active learning, especially ALcs, significantly reduces the number of required human labels while achieving comparable or even better performance than training on a much larger, randomly selected dataset.
  • Reduced Annotation Burden: The proposed methods offer a practical solution for reducing the costly and time-consuming process of annotating speech data for emotion recognition.

Key Results

  • ALcs Performance: Without instance upsampling, ALcs achieved a peak Unweighted Average Recall (UAR) of approximately 62.5% by labeling only 2.4k instances. This is about a 2% absolute gain in UAR compared to a classifier trained on the entire pool (around 10k instances), while reducing the training set size by approximately 70%.
  • ALcv Performance: With instance upsampling, ALcv achieved a UAR of 67.7% after adding 5k instances, matching a baseline performance from a previous challenge with a significantly reduced training set (4% of the total size).
  • Outperforming Passive Learning: Both ALcs and ALcv consistently outperformed passive learning (random instance selection) in terms of UAR and stability, especially in the early stages of learning.
  • Model Stability: ALcv, in particular, showed more stable performance than passive learning, indicating a more reliable learning process.

Limitations and Future Work

  • Performance Degradation: The paper notes that after reaching peak performance, ALcs's accuracy sometimes slightly decreased. This is attributed to the limited number of "sparse" (negative) instances available in the candidate pool.
  • Future Work:
    • Investigating the robustness and universality of these active learning approaches on other subjective speech tasks.
  • Exploring the potential of semi-supervised learning in conjunction with label uncertainty.

Summary updated

BibTeX
@inproceedings{zhang2013active,
  author = {Zixing Zhang and Jun Deng and Erik Marchi and Björn Schuller},
  title = {{Active learning by label uncertainty for acoustic emotion recognition}},
  booktitle = {Interspeech},
  year = {2013},
}
Figure