Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligence

Weixiang Zhao, Zhuojun Li, Shilong Wang, Yang Wang, Yulin Hu, Yanyan Zhao, Chen Wei, Bing Qin

TL;DR

Current Large Language Models (LLMs) often suffer from "catastrophic forgetting," losing general reasoning abilities when fine-tuned for emotional intelligence. This paper introduces MoEI (Modular Emotional Intelligence), a parameter-efficient framework that uses a mixture of LoRA experts and a specialized routing mechanism to significantly boost an LLM's emotional depth across perception, cognition, and expression without sacrificing its general intelligence.

What Problem Does This Paper Address?

While LLMs like GPT-4 are incredibly "smart," they are often described as emotionally stunted. Researchers have tried to fix this by fine-tuning models on emotional tasks, but these efforts usually face two major hurdles:

  1. Narrow Scope: Most work focuses only on emotion perception (e.g., sentiment analysis). True Emotional Intelligence (EI) also requires emotion cognition (understanding why someone feels a certain way) and emotion expression (responding with empathy).
  2. Catastrophic Forgetting: Naive fine-tuning on emotional data often causes the model to forget its General Intelligence (GI). As shown in the figure below, improving EI through standard methods typically leads to a sharp decline in world knowledge and reasoning.
Figure 1: Comparison of EI and GI before (in orange) and after (in blue) the EI-enhancement via naive fine- tuning based on the Flan-T5-XL (3B) backbone.

The authors argue that for an AI assistant to be truly useful, it must possess both high GI and high EI simultaneously.

Methods

The authors propose a two-part solution: a comprehensive benchmark and a novel architectural framework.

1. EIBENCH

To move beyond simple sentiment classification, the authors curated EIBENCH, a massive collection of 15 tasks across 88 datasets, all converted into a standardized text-to-text format with human-written instructions. It covers the three pillars of EI:

  • Perception: Detecting and deciphering user emotions.
  • Cognition: Reasoning about emotional causes and complex relationships (e.g., irony, metaphors).
  • Expression: Generating empathetic and supportive responses.

2. MoEI (Modular Emotional Intelligence)

The core technical contribution is the MoEI framework, which prevents the "EI vs. GI" tug-of-war through two main techniques:

  • Modular Parameter Expansion (MoLoRA): Instead of updating the entire model, MoEI adds a set of small, trainable "experts" using Low-Rank Adaptation (LoRA). Specifically, it uses a Mixture-of-LoRA architecture where multiple LoRA blocks are added to the attention and feed-forward layers.
  • Intra-Inter Modulation (I2M): A trainable router manages how these modules interact.
  • Intra-modulation determines the weighted combination of EI experts needed for a specific emotional task.
  • Inter-modulation acts as a safeguard for General Intelligence. When the model receives a GI-related input (like a math problem), the router directs the flow primarily through the frozen, original LLM backbone, minimizing the influence of the new EI modules. This is enforced during training using a KL divergence loss against replayed GI data.
Figure 3: The overall architecture of our proposed MoEI framework, which consists of two techniques, modular parameter expansion and intra-inter modulation. Red and blue lines represent the forward flow of the EI- and GI related inputs that participate in the Intra- and Inter- Modulation, respectively.

Datasets

  • EI Training/Eval: EIBENCH (15 tasks, 88 datasets including GoEmotions, EmpatheticDialogues, and SARC).
  • GI Evaluation: Standard benchmarks including MMLU (World Knowledge), Big-Bench Hard (General Reasoning), PIQA (Commonsense Reasoning), and BoolQ (Reading Comprehension).
  • Cross-Task Eval: EQ-Bench, a specialized benchmark for predicting emotional intensity in dialogues.
  • GI Replay Data: A subset of the Flan collection.

Key Insights and Ideas

  • Modularity is the Shield: By isolating emotional knowledge into specific LoRA modules and using a router to "bypass" them for general tasks, the model avoids the interference that causes catastrophic forgetting.
  • EI is Multi-Faceted: The paper demonstrates that perception, cognition, and expression are distinct enough that a model needs broad exposure (via EIBENCH) to be truly emotionally competent.
  • Small Changes, Big Impact: The authors found that relatively small parameter increments (low-rank LoRAs) are sufficient to significantly boost EI, suggesting that LLMs already have a latent "emotional foundation" that just needs proper alignment.
  • Synergy with EmotionPrompt: The study shows that EI-enhanced models are even better at utilizing "emotional stimuli" in prompts (e.g., adding "This is very important to my career") to improve performance on unrelated downstream tasks.

Key Results

The authors tested MoEI on Flan-T5 (3B and 11B) and LLaMA-2-Chat (7B and 13B).

  • EI Superiority: MoEI consistently outperformed naive fine-tuning (FT) and standard LoRA across all three EI categories. For example, on Flan-T5-XL, MoEI achieved an Emo.Exp score of 25.02, compared to 20.81 for naive fine-tuning.
  • GI Preservation: While naive fine-tuning caused LLaMA-2-Chat's General Reasoning (GR) score to plummet from 33.56 to 14.07, MoEI actually improved it slightly to 35.56.
  • Efficiency: MoEI achieved better GI preservation using only 5,000 replayed samples than baseline methods did with 100,000 samples.
Figure 4: Results of EI and GI of different methods on the larger Flan-T5-XXL (11B) backbone.

Limitations and Future Work

  • Data Accessibility: The method assumes access to the model's original training data (or a good proxy) for the "replay" part of inter-modulation, which isn't possible for closed-source models like GPT-4.
  • Task Identification: Currently, MoEI requires identifying whether a task is EI- or GI-related during training to set the modulation strategy. Future work could look into fully automated, task-agnostic routing.
  • Compute Constraints: The study was limited to models up to 13B parameters; testing on 70B+ models remains a future goal.

Summary updated

Abstract

Emotional Intelligence (EI), consisting of emotion perception, emotion cognition and emotion expression, plays the critical roles in improving user interaction experience for the current large language model (LLM) based conversational general AI assistants. Previous works mainly focus on raising the emotion perception ability of them via naive fine-tuning on EI-related classification or regression tasks. However, this leads to the incomplete enhancement of EI and catastrophic forgetting of the general intelligence (GI). To this end, we first introduce \textsc{EiBench}, a large-scale collection of EI-related tasks in the text-to-text formation with task instructions that covers all three aspects of EI, which lays a solid foundation for the comprehensive EI enhancement of LLMs. Then a novel \underline{\textbf{Mo}}dular \underline{\textbf{E}}motional \underline{\textbf{I}}ntelligence enhancement method (\textbf{MoEI}), consisting of Modular Parameter Expansion and intra-inter modulation, is proposed to comprehensively enhance the EI of LLMs without compromise their GI. Extensive experiments on two representative LLM-based assistants, Flan-T5 and LLaMA-2-Chat, demonstrate the effectiveness of MoEI to improving EI while maintain GI.
BibTeX
@inproceedings{zhao2024both,
  author = {Weixiang Zhao and Zhuojun Li and Shilong Wang and Yang Wang and Yulin Hu and Yanyan Zhao and Chen Wei and Bing Qin},
  title = {{Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligence}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.665},
}
Figure