Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective

Yiqun Zhang, Xiaocui Yang, Xingle Xu, Zeran Gao, Yijie Huang, Shiyi Mu, Shi Feng, Daling Wang, Yifei Zhang, Kaisong Song, Ge Yu

Abstract

Affective Computing (AC), integrating computer science, psychology, and cognitive science knowledge, aims to enable machines to recognize, interpret, and simulate human emotions. Affective Computing (AC) includes two mainstream tasks: Affective Understanding (AU) and Affective Generation (AG). Fine-tuning Pre-trained Language Models (PLMs) for AU tasks has succeeded considerably; however, these models lack generalization ability. The emergence of Large Language Models (LLMs), such as the ChatGPT series and LLaMA models, brings new opportunities and challenges, catalyzing a paradigm shift in AC. LLMs possess capabilities of in-context learning, common sense reasoning, and advanced sequence generation. This survey summarizes the development of LLM research in this field, outlining techniques like Instruction Tuning and Prompt Engineering, and summarizing existing benchmarks and evaluation methods.

Introduction

For decades, Affective Computing relied on "small" pre-trained models (like BERT or RoBERTa) that were fine-tuned on narrow, manually labeled datasets. While these models were effective for simple sentiment analysis, they were often "one-trick ponies" that struggled to generalize to new domains or generate nuanced, empathetic responses.

The rise of Large Language Models (LLMs) has fundamentally changed this landscape. As shown in Figure 1, the field is moving from task-specific datasets and models toward multi-task datasets and general-purpose LLMs that can be adapted through prompting and instruction tuning.

The paradigm shift from task-specific PLMs to multi-task LLMs in Affective Computing.

Technical Pillars and Taxonomy

The adaptation of LLMs for AC is organized into three primary technical pillars: Instruction Tuning, Prompt Engineering, and Reinforcement Learning. Figure 2 provides a comprehensive taxonomy of the current research landscape.

A comprehensive taxonomy of Affective Computing research categorized by methodology and task.

1. Instruction Tuning

This involves fine-tuning models on datasets where tasks are explicitly defined by natural language instructions. Researchers often utilize Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and Prompt-Tuning. These techniques allow models to learn emotional nuances and acquire "affect-aware" capabilities without the prohibitive cost of updating billions of parameters.

2. Prompt Engineering

Beyond simple zero-shot queries, advanced strategies have emerged to unlock the latent emotional intelligence of LLMs:

  • Chain-of-Thought (CoT): Forcing the model to "think out loud" about emotional triggers before providing a label.
  • Agent-based Prompting: Assigning the LLM a persona (e.g., "an empathetic therapist") or using multi-agent systems where different "experts" debate a user's emotional state.

3. Reinforcement Learning (RL)

RL is used to align models with human emotional preferences. This includes RLHF (Human Feedback), RLAIF (AI Feedback), and RLVR (Verifiable Rewards), where models are rewarded for generating responses that are polite, empathetic, and non-toxic.

Key Insights and Performance

  • From Labeling to Reasoning: In the pre-LLM era, AC was primarily a classification problem. Now, it is a reasoning problem. Techniques like Three-hop Reasoning (THOR) allow models to identify an aspect, determine sentiment, and explain the underlying cause.
  • The Empathy Gap: While LLMs like GPT-4 can sometimes exceed human performance in empathy levels (by roughly 10% in some studies), they still struggle with highly structured tasks like Aspect-Based Sentiment Analysis (ABSA) compared to smaller, specialized models.
  • Instruction Tuning Efficacy: Applying LoRA or P-Tuning can improve performance across multiple affective datasets simultaneously, suggesting that emotional intelligence is a transferable skill within the model's weights.

Benchmarks and Evaluation

Evaluation has moved beyond traditional sentiment analysis staples like Yelp and IMDb. Newer benchmarks include:

  • SOUL (Sentiment and Opinion Understanding of Language): Focuses on complex linguistic nuances.
  • EQ-Bench: Specifically tests emotional intelligence in dialogue.
  • Multimodal Sets: Benchmarks like MERBench and DFEW integrate video and audio cues alongside text, acknowledging that true emotional intelligence requires "seeing" and "hearing."

Limitations and Future Work

Despite the progress, several challenges remain. Current reward functions for RL often lack grounding in deep psychological theories, leading to "mechanical" empathy. Furthermore, capturing how emotions evolve over long, multi-turn conversations (temporal modeling) remains an open challenge. Future research must also address the ethical and privacy concerns inherent in processing deeply personal affective data.

Summary updated

Abstract

Affective Computing (AC), integrating computer science, psychology, and cognitive science knowledge, aims to enable machines to recognize, interpret, and simulate human emotions.To create more value, AC can be applied to diverse scenarios, including social media, finance, healthcare, education, etc. Affective Computing (AC) includes two mainstream tasks, i.e., Affective Understanding (AU) and Affective Generation (AG). Fine-tuning Pre-trained Language Models (PLMs) for AU tasks has succeeded considerably. However, these models lack generalization ability, requiring specialized models for specific tasks. Additionally, traditional PLMs face challenges in AG, particularly in generating diverse and emotionally rich responses. The emergence of Large Language Models (LLMs), such as the ChatGPT series and LLaMA models, brings new opportunities and challenges, catalyzing a paradigm shift in AC. LLMs possess capabilities of in-context learning, common sense reasoning, and advanced sequence generation, which present unprecedented opportunities for AU. To provide a comprehensive overview of AC in the LLMs era from an NLP perspective, we summarize the development of LLMs research in this field, aiming to offer new insights. Specifically, we first summarize the traditional tasks related to AC and introduce the preliminary study based on LLMs. Subsequently, we outline the relevant techniques of popular LLMs to improve AC tasks, including Instruction Tuning and Prompt Engineering. For Instruction Tuning, we discuss full parameter fine-tuning and parameter-efficient methods such as LoRA, P-Tuning, and Prompt Tuning. In Prompt Engineering, we examine Zero-shot, Few-shot, Chain of Thought (CoT), and Agent-based methods for AU and AG. To clearly understand the performance of LLMs on different Affective Computing tasks, we further summarize the existing benchmarks and evaluation methods.
BibTeX
@misc{zhang2024affective,
  author = {Yiqun Zhang and Xiaocui Yang and Xingle Xu and Zeran Gao and Yijie Huang and Shiyi Mu and Shi Feng and Daling Wang and Yifei Zhang and Kaisong Song and Ge Yu},
  title = {{Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective}},
  year = {2024},
  eprint = {2408.04638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.04638},
}
Figure