Distribution-Based Emotion Recognition in Conversation

Wen Wu, Chao Zhang, Philip C. Woodland

Abstract

Emotion Recognition in Conversation (ERC) is traditionally treated as a hard classification task, often discarding ambiguous data and ignoring the inherent subjectivity of human emotions. This paper introduces a distribution-based framework that utilizes Dirichlet Prior Networks (DPN) within a Transformer-based sequence-to-sequence architecture. By modeling emotions as probability distributions rather than single labels, the system captures uncertainty and utilizes previously discarded "no-agreement" utterances. Evaluated on the IEMOCAP dataset, the proposed DPN-KL model demonstrates superior performance in both classification accuracy and uncertainty estimation, highlighting the importance of modeling emotional ambiguity in conversational AI.

Introduction

Recognizing emotions within a dialogue is a complex challenge that extends beyond simple sentiment analysis of isolated sentences. Human emotions are deeply context-dependent, persistent over time, and fundamentally subjective. Traditional ERC models typically rely on "majority voting," where multiple human annotators label an utterance, and only the label with the highest agreement is retained.

This standard approach suffers from two critical flaws. First, it leads to significant data waste; utterances where annotators disagree are often discarded, which breaks the temporal flow of the dialogue context. Second, it results in a loss of nuance. Forcing a single label onto an utterance removes the inherent uncertainty of human perception. For instance, if a subset of annotators perceives "frustration" while others perceive "sadness," the true emotional state is likely a blend of both rather than a binary choice. This research proposes a shift toward a distribution-based approach that embraces this ambiguity as a feature rather than a limitation.

Methodology

The researchers from the University of Cambridge propose a sequence-to-sequence framework designed to model a dialogue as a continuous series of emotion distributions.

Multimodal Architecture

The system utilizes a Transformer encoder-decoder architecture. To represent the richness of human communication, the model extracts features from two state-of-the-art self-supervised learning (SSL) models:

  • Wav2Vec 2.0 for capturing acoustic and prosodic information from audio.
  • BERT for capturing semantic meaning from text.

These features are integrated using bilinear pooling with shortcut connections. This fusion method allows the model to capture complex, non-linear interactions between the verbal content and the vocal delivery of an utterance.

Dirichlet Prior Networks (DPN)

At the heart of the framework is the use of Dirichlet Prior Networks. Instead of predicting a standard softmax probability, the model predicts the parameters of a Dirichlet distribution. This Bayesian approach allows the system to learn an "utterance-specific prior," enabling it to distinguish between different types of uncertainty—specifically, whether the model itself is confused or if the underlying data is inherently ambiguous.

Training and Sequence Modeling

To address the challenge of "exposure bias"—where a model becomes overly reliant on perfect historical data during training—the authors implemented scheduled sampling. This technique forces the model to process its own previous, potentially erroneous predictions during training, making it more robust during real-world inference. Furthermore, the sequence-to-sequence formulation ensures that the emotion of any given utterance is conditioned on the entire preceding conversational history.

Experimental Setup and Results

The study utilized the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset, consisting of approximately 10,000 utterances across 151 dialogues. A key advantage of the distribution-based method is its ability to utilize the 45% of data typically discarded in standard 4-way classification tasks due to lack of majority agreement. By maintaining a contiguous context, the model preserves the "memory" of the dialogue, which is essential for tracking emotional shifts.

Performance Metrics

The proposed DPN-KL system (which combines Dirichlet loss with Kullback-Leibler divergence) was benchmarked against standard "HARD" classification and "SOFT" label smoothing baselines:

  • Classification Accuracy: In a 5-way classification task (including an "others" category), the DPN-KL system achieved a Weighted Accuracy (WA) of 63.63%, significantly outperforming the HARD baseline of 59.99%.
  • Uncertainty Estimation: Using the Area Under the Precision-Recall curve (AUPR) to measure the identification of ambiguous utterances, the DPN-KL model reached 81.17%. This indicates a superior ability to identify when an emotional state is unclear compared to traditional models.
  • Fusion Impact: The integration of multimodal data proved vital; combining audio (W2V2) and text (BERT) resulted in a performance jump from approximately 70% WA to 74.8% in baseline configurations.

Discussion and Limitations

While the distribution-based approach marks a significant advancement in creating emotionally intelligent AI, several limitations remain. The current study focuses on dyadic (two-person) conversations, whereas real-world applications often involve multi-party interactions. Additionally, the computational overhead of running simultaneous Wav2Vec 2.0 and BERT models may pose challenges for deployment on edge devices or in low-latency environments. Finally, the model still requires multiple human annotations per utterance to establish a ground-truth distribution, a requirement that is not met by many existing datasets.

Conclusion

By treating emotions as distributions rather than discrete categories, this framework provides a more realistic representation of human communication. The ability to model uncertainty and maintain conversational context allows for higher accuracy and better generalization, moving the field closer to AI that truly understands the "grey areas" of human emotion.

Summary updated

BibTeX
@inproceedings{wu2022distribution,
  author = {Wen Wu and Chao Zhang and Philip C. Woodland},
  title = {{Distribution-Based Emotion Recognition in Conversation}},
  booktitle = {SLT},
  year = {2022},
  url = {https://openreview.net/forum?id=RGpBW5DDRZ},
}
Figure