PFA-ERC: Psuedo-Future Augmented Dynamic Emotion Recognition in Conversations

Tanmay Khule, Rishabh Agrawal, Apurva NarayanEMNLP 2024 · OpenReview

Abstract

AI systems’ ability to interpret human emotions and adapt to variations is becoming more crucial as AI gets embedded into everyone’s daily lives. Emotion Recognition in Conversations (ERC) is based on this fundamental challenge. Current state-of-the-art technologies in ERC are limited due to the need for future information. We introduce High-Dimensional Temporal Fusion Transformer (HiTFT), a time-series forecasting transformer that predicts pseudo-future information to overcome this constraint. This retains the models’ dynamic nature and provides future information more efficiently than other methods. Our proposed method combines pseudo future embeddings with an encoder that models the speaker’s emotional state using past and pseudo-future information as well as inter and intra speaker interactions; these speaker states are then passed through a decoder block that predicts the inferred emotion of that utterance. We further evaluate our method and show that it achieves state of the art performance on three ERC datasets - MELD, EmoryNLP, and IEMOCap.

Introduction

Emotion Recognition in Conversations (ERC) is a critical task for developing empathetic AI. However, a significant challenge in real-time ERC is the lack of future context. In human dialogue, the emotional weight of an utterance is often clarified by subsequent responses. For instance, a neutral statement might be reinterpreted as sarcastic based on a listener's later reaction. While existing state-of-the-art models leverage future context, they typically rely on recorded transcripts (offline) or use computationally expensive Large Language Models (LLMs) to "hallucinate" future text.

We propose PFA-ERC (Pseudo-Future Augmented ERC), a framework that treats dialogue as a time-series forecasting problem. Instead of predicting future text, PFA-ERC predicts future mathematical embeddings—termed "pseudo-futures"—enabling real-time performance with significant improvements in speed and memory efficiency.

Methodology

The PFA-ERC architecture is designed to capture emotional trajectories without the overhead of text generation. The process involves four key stages:

  1. Feature Extraction: Utterances are converted into high-dimensional vector representations using a pre-trained RoBERTa-Large model.
  2. Pseudo-Future Extraction (HiTFT): We introduce the High-Dimensional Temporal Fusion Transformer (HiTFT), adapted from time-series forecasting models. HiTFT forecasts the embeddings of the next four utterances based on historical context.
  3. Speaker Emotion Encoder: This module utilizes attention mechanisms to model both "intra-speaker" emotional states (a speaker's own trajectory) and "inter-speaker" interactions (how speakers influence each other).
  4. Decoder: A Gated Recurrent Unit (GRU) integrates the current utterance embedding, the modeled speaker states, and the predicted pseudo-future embeddings to produce the final emotion classification.
The PFA-ERC architecture, featuring RoBERTa for feature extraction, HiTFT for pseudo-future prediction, and specialized encoders for speaker modeling.

Experimental Setup

The model was rigorously evaluated on three benchmark ERC datasets:

  • IEMOCAP: Professional actor dialogues with an average of 52 utterances per conversation.
  • MELD: Short dialogues from the TV show Friends, averaging 9 utterances.
  • EmoryNLP: A Friends-based dataset featuring complex, multi-party emotional labels.

Results and Discussion

PFA-ERC achieves state-of-the-art (SOTA) performance across multiple benchmarks, particularly excelling in longer dialogues where temporal patterns are more pronounced.

Comparative performance of PFA-ERC against existing ERC models across MELD, IEMOCAP, and EmoryNLP datasets.

Key Insights

  • Dialogue as a Time-Series: Treating conversation as a multivariate time-series signal allows the model to bypass the linguistic "bottleneck" of text generation.
  • The "Smoothness" Advantage: Spectral analysis reveals that pseudo-future embeddings are cleaner and more stable than real future embeddings. By forecasting trends rather than specific words, the model effectively denoises the conversational signal.
  • Efficiency: PFA-ERC is 3x faster than LLM-based competitors like InstructERC and requires 50% less memory (12GB vs 28GB VRAM), making it suitable for deployment on edge hardware.

Conclusion and Future Work

While PFA-ERC sets new benchmarks on IEMOCAP and EmoryNLP, it shows more modest gains on the MELD dataset due to the brevity of its dialogues, which limits the historical context available for time-series forecasting. Future work will explore dynamic "look-ahead" windows and the integration of multimodal cues (audio and visual) to further enhance emotional forecasting.

Summary updated

Abstract

AI systems’ ability to interpret human emotions and adapt to variations is becoming more crucial as AI gets embedded into everyone’s daily lives. Emotion Recognition in Conversations (ERC) is based on this fundamental challenge. Current state-of-the-art technologies in ERC are limited due to the need for future information. We introduce High-Dimensional Temporal Fusion Transformer (HiTFT), a time-series forecasting transformer that predicts pseudo-future information to overcome this constraint. This retains the models’ dynamic nature and provides future information more efficiently than other methods. Our proposed method combines pseudo future embeddings with an encoder that models the speaker’s emotional state using past and pseudo-future information as well as inter and intra speaker interactions; these speaker states are then passed through a decoder block that predicts the inferred emotion of that utterance. We further evaluate our method and show that it achieves state of the art performance on three ERC datasets - MELD, EmoryNLP, and IEMOCap.
BibTeX
@inproceedings{khule2024pfa,
  author = {Tanmay Khule and Rishabh Agrawal and Apurva Narayan},
  title = {{PFA-ERC: Psuedo-Future Augmented Dynamic Emotion Recognition in Conversations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=AnB9kE3MtG},
}
Figure