Agglomerative hierarchical clustering of emotions in speech based on subjective relative similarity

Ryoichi Takashima, Tohru Nagano, Ryuki Tachibana, Masafumi NishimuraInterspeech 2011

TL;DR

This paper introduces a novel method for clustering speech emotions using a new type of subjective similarity data called "ABC data." This approach allows for more accurate hierarchical clustering by learning feature weights that adapt to different layers of the hierarchy, mimicking human perception and significantly reducing classification errors in call center scenarios.

What Problem Does This Paper Address?

Call center analytics systems aim to understand the emotions conveyed in customer and agent speech. While classifying emotions into predefined categories like "happy" or "sad" is common, it's challenging to define these categories definitively and they may not capture the subtle nuances of human perception. Humans often perceive emotions in a hierarchical and relative manner – for example, distinguishing between "thinking" and "mildly dissatisfied" within a broader "calm and negative" category. Traditional hierarchical clustering methods struggle to incorporate subjective similarity data effectively, especially when the number of desired clusters isn't known beforehand. Existing methods for incorporating subjective similarity, like Must-Link/Cannot-Link constraints or XAB relative similarity data, have limitations when applied to hierarchical clustering, making it difficult to optimize feature weights and evaluate clustering performance consistently.

Methods

The core innovation is the introduction of ABC data, a new type of subjective similarity data. Instead of asking labelers to compare two samples (XAB data: "X is closer to A than X is to B"), ABC data asks labelers to identify the most similar pair within a set of three samples (A, B, C). This provides richer information, effectively giving two relative similarity inequalities (e.g., Sim(A, B) > Sim(B, C) and Sim(A, B) > Sim(C, A)).

This ABC data is used in a closed-loop semi-supervised clustering algorithm. The algorithm has two phases:

  1. Learning Phase: This phase iteratively refines feature weights. In each iteration, unsupervised agglomerative hierarchical clustering (specifically Ward's method) is performed on the training data using the current feature weights. The clustering results are then evaluated using the ABC training data. Based on this evaluation, feature weights are updated using an AdaBoost-like algorithm, where weights are increased for data points that were misclassified. This process continues until feature weights are optimized.
  2. Runtime Phase: Once feature weights are learned, the same unsupervised hierarchical clustering algorithm is applied to the test data using these optimized weights.

A key enhancement is the ability to learn different sets of weights for different layers of the clustering hierarchy. This is inspired by human perception, where different features are prioritized when distinguishing between very different emotions versus very similar ones. The algorithm heuristically determines the number of layers and learns a specific set of weights for each layer, allowing for more nuanced emotional representation.

Datasets

The paper uses a custom dataset collected from a real-life Japanese call center.

  • Corpus: Utterances of a specific sentence, "Sou desu ne" (meaning "That's right"), spoken by a female agent. This sentence is chosen because its emotional conveyance varies significantly with subtle delivery.
  • Data Collection: 67 utterances were extracted. 33 were used for training and 34 for testing.
  • ABC Relative Similarity Data: This new benchmark data was created by asking 3 human labelers to mark the most similar pair within randomly selected triplets of utterances.
  • 100 tuples for learning.
  • 250 tuples for testing.

Key Insights and Ideas

  • ABC Data Superiority: The new ABC data format is more suitable for hierarchical clustering than traditional XAB data because it doesn't require pre-defining the number of clusters and provides richer, more consistent information for both learning and evaluation.
  • Mimicking Human Perception: The ability to learn different feature weights for different layers of the hierarchy is crucial. It allows the model to adapt its focus, prioritizing broad features for initial distinctions and finer features for subtle differences, mirroring how humans perceive emotional nuances.
  • Closed-Loop Optimization: Integrating clustering evaluation directly into the feature weight learning loop ensures that the weights are optimized for the specific task of hierarchical clustering, leading to better performance.
  • "Sou desu ne" as a Rich Emotional Carrier: The choice of this specific sentence highlights how even seemingly simple phrases can carry a wide range of attitudes and intentions, making them a challenging yet valuable testbed for emotion recognition.

Key Results

The proposed method, especially when using multiple sets of feature weights (Proposed(L)), demonstrated significant improvements:

  • Reduced Error Rate: Proposed(L) reduced the error rate by 15.2% compared to the baseline PCA method (which did not use ABC data).
  • Comparison to Baseline Methods:
    • Proposed (single set of weights) achieved a 36.4% error rate, outperforming Baseline (40.3%) and PCA (38.8%).
  • FS (Forward Selection) actually increased the error rate to 41.5% in Table 1, suggesting its greedy approach was less effective.
  • Impact of Layered Weights: Using multiple layers of weights (Proposed(L) at 32.9% error) significantly improved performance over a single set of weights (Proposed at 36.4% error). FS also benefited from layered weights (FS(L) at 35.3% error).
  • Statistical Significance: The improvements for Proposed(L) over Proposed and PCA were statistically significant (P < 0.03, Chi-square test).

Limitations and Future Work

  • Speaker-Independent Clustering: The current work focuses on a single female speaker. The authors acknowledge that future work needs to address speaker-independent clustering of arbitrary utterances.
  • Feature Set: While the paper uses a set of 16 features (prosodic and spectral), it mentions that a richer feature set, including linguistic features, could further improve classification accuracy.
  • Sentence Specificity: The experiments were conducted on a single, specific sentence. Generalizing to a wider range of utterances and emotional expressions would be a logical next step.

Summary updated

BibTeX
@inproceedings{takashima2011agglomerative,
  author = {Ryoichi Takashima and Tohru Nagano and Ryuki Tachibana and Masafumi Nishimura},
  title = {{Agglomerative hierarchical clustering of emotions in speech based on subjective relative similarity}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure