FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning

Zhuozhao Hu, Kaishen Yuan, Xin Liu, Zitong Yu, Yuan Zong, Jingang Shi, Huanjing Yue, Jingyu Yang

Abstract

Facial Emotion Analysis (FEA) is a critical component of human-computer interaction, yet current Multimodal Large Language Models (MLLMs) often struggle with the fine-grained visual cues necessary for accurate and interpretable emotion recognition. We present FEALLM, a novel framework that bridges the gap between coarse facial expression recognition (FER) and fine-grained action unit detection (AUD). By introducing a specialized instruction-tuning dataset and an architecture designed to capture subtle local facial details, FEALLM transforms emotion recognition from a simple labeling task into a transparent reasoning process. Our results demonstrate significant improvements in accuracy, interpretability, and generalization across multiple benchmarks.

Introduction

While artificial intelligence has made significant strides in labeling facial expressions—categorizing a face as "happy," "sad," or "angry"—these systems often operate as "black boxes." They lack the interpretability required for high-stakes applications and frequently fail to generalize across diverse lighting conditions or ethnicities. Scientifically, emotions are not monolithic labels but the result of coordinated muscle movements known as Action Units (AUs). For instance, a "Duchenne smile" involves both the zygomatic major muscle (lip corner puller) and the orbicularis oculi muscle (cheek raiser).

Traditional models typically treat Facial Expression Recognition (FER) and Action Unit Detection (AUD) as isolated tasks. Furthermore, general-purpose MLLMs like LLaVA or GPT-4 often overlook the subtle, low-level visual cues—such as a slight squint or a tiny twitch of the lip—that are essential for distinguishing between genuine and feigned emotions.

Comparison of FEALLM with traditional and general-purpose MLLM approaches for facial emotion analysis.

Methodology

FEALLM addresses these limitations through three primary architectural and data-driven innovations:

1. FEA Instruction Dataset

The researchers constructed a new FEA Instruction Dataset using the Aff-Wild2 dataset as a foundation. By leveraging GPT-4o, they generated structured "Chain-of-Thought" descriptions for nearly 15,000 images. Each entry includes an Emotion Summary, a Facial Movement Description (detailing specific AUs), and Emotion Reasoning that establishes the causal link between muscle movements and the resulting emotion.

2. Local Clue Aggregator (LCA)

To prevent the loss of subtle details during the encoding process, the LCA crops the face into 16 local regions at multiple scales. This allows the model to extract high-resolution local features—such as the specific fold of a brow or the tension in a lip—that are typically discarded by general-purpose vision encoders.

3. Multi-perspective Projector (MPP)

Standard vision encoders prioritize high-level semantic features. However, in emotion analysis, "shallow" features like edges and textures are paramount. The MPP extracts these shallow features from earlier layers of the vision encoder, providing the model with the raw visual evidence needed to detect fine muscle movements.

Example of the FEA Instruction format, showcasing the reasoning process from Action Units to final emotion label.

Datasets and Evaluation

The model was trained on the FEA Instruction Dataset, which contains 14,892 images covering 7 basic emotions and 12 Action Units. To rigorously evaluate performance, the authors introduced FEABench, a benchmark of 1,335 images designed to test both expression recognition and muscle movement detection simultaneously.

To assess generalization, FEALLM was subjected to zero-shot evaluation on several external datasets:

  • FER: RAF-DB and AffectNet.
  • AUD: BP4D and DISFA.

Key Insights: The Power of Emotional Synergy

A central finding of this work is that FER and AUD are mutually beneficial. By forcing the model to explain why it chooses an emotion label through instruction tuning, the model becomes more robust. This "reasoning as a regularizer" approach prevents the model from relying on spurious correlations or biases in the training data. Furthermore, the results confirm that shallow visual features are deeply important for emotion analysis; the specific lines and wrinkles associated with muscle movement are the "final answer" in this domain.

Results

FEALLM significantly outperformed both general-purpose MLLMs and specialized state-of-the-art models:

  • Accuracy: On FEABench, FEALLM achieved 67.36% accuracy in expression recognition, compared to 59.97% for a fine-tuned LLaVA-1.5 and 45.71% for Qwen-VL.
  • Action Unit Detection: The model achieved an average F1 score of 59.60%, nearly doubling the performance of most zero-shot MLLMs.
  • Generalization: In zero-shot tests on the RAF-DB dataset, FEALLM reached 69.95% accuracy, demonstrating that its reasoning capabilities transfer effectively to unseen environments and facial types.

Conclusion and Future Work

FEALLM represents a significant step toward more transparent and accurate facial emotion analysis. By linking high-level labels to low-level muscle movements, it provides a framework for AI that can truly "read between the lines" of a human face. Future research will focus on extending this framework to dynamic video and audio data to capture the temporal evolution of emotions, as well as expanding the dataset to include more diverse "in-the-wild" scenarios for improved real-world robustness.

Summary updated

BibTeX
@inproceedings{hu2025feallm,
  author = {Zhuozhao Hu and Kaishen Yuan and Xin Liu and Zitong Yu and Yuan Zong and Jingang Shi and Huanjing Yue and Jingyu Yang},
  title = {{FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning}},
  booktitle = {ACM Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=nN8J2iUEBA},
}
Figure