A cluster-profile representation of emotion using agglomerative hierarchical clustering

Emily Mower, Kyu Jeong Han, Sungbok Lee, Shrikanth S. NarayananInterspeech 2010 · DOI

Introduction

Automatic emotion recognition typically relies on supervised learning, which requires massive amounts of human-labeled data. However, labeling emotion is notoriously difficult: it is expensive, time-consuming, and highly subjective. Furthermore, human emotion is rarely "pure"; an utterance might be a subtle mix of frustration and sadness, making a single categorical label (e.g., "Angry") an oversimplification.

Previous work used Emotion Profiles (EP), where an utterance is described by its similarity to four basic categories: anger, happiness, neutrality, and sadness. While effective, EPs still require those specific categories to be labeled in the training set. This paper introduces Cluster-Profiles (CP), a novel way to represent human emotion by grouping data into unsupervised clusters rather than relying solely on predefined semantic labels.

Methodology

The authors propose a system-level semi-supervised approach consisting of four main stages:

  1. Unsupervised Feature Selection: They use Principal Feature Analysis (PFA)—an extension of PCA—to narrow down 685 initial features (from audio and motion capture) to the 20 most representative ones.
  2. Unsupervised Clustering: They apply Agglomerative Hierarchical Clustering (AHC) to the unlabeled training data. This groups utterances based on their inherent similarities in the feature space rather than their emotional labels.
  3. Profile Generation: For each discovered cluster, they train a binary Support Vector Machine (SVM) (Cluster vs. Everything Else). When a new utterance comes in, it is passed through these $n$ SVMs. The distances to the decision hyperplanes form an $n$-dimensional vector called the Cluster-Profile (CP).
  4. Supervised Classification: Finally, a Naive Bayes classifier is trained on these CPs to map them to the final target emotion labels.
System architecture for generating and classifying Cluster-Profiles (CP).

This approach is "semi-supervised" because the feature selection and clustering don't need labels; only the final classification step requires a small set of labeled data from the target speaker.

Experimental Setup

The study uses the USC IEMOCAP Database, a well-known benchmark in affective computing.

  • Content: Dyadic interactions between actors performing scripted and improvised scenarios.
  • Modalities: Audio, video, and motion capture (facial markers, head position, and hand movements).
  • Scale: 4,806 utterances across 10 speakers.
  • Labels: The study focuses on the four most common: Angry, Happy, Neutral, and Sad.

Key Results and Insights

The CP-based system achieved a weighted accuracy of 69.25%, slightly outperforming the label-based EP baseline of 68.37%. As shown in the performance table, the best results were generally achieved with 11 to 15 clusters.

Performance comparison (F-measure and Accuracy) between the EP baseline and CP models with varying cluster counts.

Data-Driven "Basis" Functions

The clusters act as a "basis" for representing emotion. The results suggest that these mathematical clusters capture affective properties just as well as—and sometimes better than—human-defined semantic categories. This proves that human-centric labels aren't strictly necessary for the intermediate representation of emotion.

The "Neutrality" Breakthrough

One of the biggest challenges in emotion AI is identifying "Neutral" states, which are often poorly defined. The CP method improved the F-measure for the Neutral class by 20% relative over the baseline (0.54 vs 0.45). This suggests that "neutrality" is easier to define as a cluster in feature space than as a semantic concept.

Conclusion

The Cluster-Profile representation demonstrates that effective emotion recognizers can be built without exhaustively labeling every second of training data. By using unsupervised clustering to create an intermediate representation, the system achieves performance parity with supervised methods while offering greater flexibility.

Future work will explore different clustering methods and investigate Collaborative Filtering to further personalize the emotion assessment without increasing the labeling burden. The authors also aim to test the robustness of this method across different datasets.

Summary updated

BibTeX
@inproceedings{mower2010cluster,
  author = {Emily Mower and Kyu Jeong Han and Sungbok Lee and Shrikanth S. Narayanan},
  title = {{A cluster-profile representation of emotion using agglomerative hierarchical clustering}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-288},
}
Figure