Emotion Transfer with Enhanced Prototype for Unseen Emotion Recognition in Conversation

Kun Peng, Cong Cao, Hao Peng, Guanlin Wu, Zhifeng Hao, Lei Jiang, Yanbing Liu, Philip S. Yu

Abstract: Current Emotion Recognition in Conversation (ERC) research follows a closed-domain assumption. However, there is no clear consensus on emotion classification in psychology, which presents a challenge for models when it comes to recognizing previously unseen emotions in real-world applications. To bridge this gap, we introduce the Unseen Emotion Recognition in Conversation (UERC) task for the first time and propose ProEmoTrans, a solid prototype-based emotion transfer framework. This prototype-based approach shows promise but still faces key challenges: First, implicit expressions complicate emotion definition, which we address by proposing an LLM-enhanced description approach. Second, utterance encoding in long conversations is difficult, which we tackle with a proposed parameter-free mechanism for efficient encoding and overfitting prevention. Finally, the Markovian flow nature of emotions is hard to transfer, which we address with an improved Attention Viterbi Decoding (AVD) method to transfer seen emotion transitions to unseen emotions. Extensive experiments on three datasets show that our method serves as a strong baseline for preliminary exploration in this new area.

Introduction: The Closed-World Problem in ERC

Emotion Recognition in Conversation (ERC) is a cornerstone for developing empathetic AI systems. However, current research predominantly operates under a "closed-domain" assumption, where models are trained and tested on a predefined, static set of emotion labels. In reality, psychological consensus on emotion classification is lacking, and real-world interactions involve a vast spectrum of emotions that may not be present in any single training dataset.

When models encounter "unseen" emotions—labels not present during training—their performance typically degrades. This gap necessitates the Unseen Emotion Recognition in Conversation (UERC) task. The primary challenges in UERC include handling implicit emotional expressions, avoiding overfitting to specific conversation flows of "seen" emotions, and transferring learned transition patterns to new emotional categories.

Emotion categories and transition patterns vary significantly across common ERC datasets like IEMOCAP, MELD, and EmoryNLP.

ProEmoTrans: A Framework for Emotion Transfer

To address these challenges, we propose ProEmoTrans, a prototype-based emotion transfer framework designed to recognize unseen emotions by leveraging knowledge from seen ones. The architecture consists of three core innovations:

1. LLM-enhanced Emotion Description (LED)

Traditional prototype-based methods often rely on dictionary definitions, which are often too sterile to capture the nuance of conversational context. ProEmoTrans utilizes Large Language Models (LLMs) like ChatGPT-3.5 to generate sentences that implicitly express target emotions. By using these context-rich descriptions, the model constructs more robust and representative prototype vectors for each emotion category.

2. Gaussian Self-Attention (GAS)

Utterance encoding in long conversations is prone to overfitting, especially when using complex modules like GCNs or LSTMs that memorize specific "seen" patterns. We introduce a parameter-free Gaussian Self-Attention mechanism. GAS weights attention based on a Gaussian distribution relative to the distance between utterances, ensuring that local context is prioritized while preventing the model from becoming overly specialized to training data.

3. Attention Viterbi Decoding (AVD)

Emotional flow is inherently Markovian (e.g., a transition from "Neutral" to "Anger"). To transfer these transition rules to unseen emotions, we adapt the Viterbi decoding algorithm. By calculating the similarity between seen and unseen emotion prototypes, AVD "stretches" the learned transition matrix, allowing the model to infer the most likely sequence of emotions even for categories it has never encountered.

The ProEmoTrans architecture integrates LLM-enhanced descriptions (LED), parameter-free Gaussian attention (GAS), and adapted Viterbi decoding (AVD).

Experimental Results

We evaluated ProEmoTrans across three major datasets: IEMOCAP, EmoryNLP, and MELD, using a cross-dataset zero-shot configuration.

  • Superior Performance: ProEmoTrans significantly outperformed traditional baselines and state-of-the-art LLMs, including GPT-4o and DeepSeek-V3. It achieved weighted F1 score improvements ranging from 2.05% to 11.58% over these models.
  • Ablation Insights: The LED module proved to be the most critical component; its removal led to a 14.3% average drop in F1 score, highlighting the importance of high-quality, context-aware prototypes.
  • Efficiency: The model maintains high inference efficiency, with processing times (6-9ms per sample) comparable to standard BERT-based architectures, making it suitable for real-time applications.

Conclusion

ProEmoTrans establishes a strong baseline for the UERC task by bridging the gap between seen and unseen emotions through prototype transfer. By combining LLM-driven semantic enrichment with parameter-free structural priors and adaptive sequence decoding, the framework demonstrates remarkable generalization capabilities. Future work will explore multi-modal integration—incorporating acoustic and visual cues—to further enhance the recognition of complex, implicit emotional states.

Summary updated

Abstract

Current Emotion Recognition in Conversation (ERC) research follows a closed-domain assumption. However, there is no clear consensus on emotion classification in psychology, which presents a challenge for models when it comes to recognizing previously unseen emotions in real-world applications. To bridge this gap, we introduce the Unseen Emotion Recognition in Conversation (UERC) task for the first time and propose **ProEmoTrans**, a solid prototype-based emotion transfer framework. This prototype-based approach shows promise but still faces key challenges: First, implicit expressions complicate emotion definition, which we address by proposing an LLM-enhanced description approach. Second, utterance encoding in long conversations is difficult, which we tackle with a proposed parameter-free mechanism for efficient encoding and overfitting prevention. Finally, the Markovian flow nature of emotions is hard to transfer, which we address with an improved Attention Viterbi Decoding (AVD) method to transfer seen emotion transitions to unseen emotions. Extensive experiments on three datasets show that our method serves as a strong baseline for preliminary exploration in this new area.
BibTeX
@inproceedings{peng2025emotion,
  author = {Kun Peng and Cong Cao and Hao Peng and Guanlin Wu and Zhifeng Hao and Lei Jiang and Yanbing Liu and Philip S. Yu},
  title = {{Emotion Transfer with Enhanced Prototype for Unseen Emotion Recognition in Conversation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=ufJ0cwWKC6},
}
Figure