ExpLLM: Towards Chain of Thought for Facial Expression Recognition

Xing Lan, Jian Xue, Ji Qi, Dongmei Jiang, Ke Lu, Tat-Seng Chua

Abstract

Facial expression recognition (FER) is a critical task in multimedia with significant implications across various domains. However, analyzing the causes of facial expressions is essential for accurately recognizing them. Current approaches, such as those based on facial action units (AUs), typically provide AU names and intensities but lack insight into the interactions and relationships between AUs and the overall expression. In this paper, we propose a novel method called ExpLLM, which leverages large language models to generate an accurate chain of thought (CoT) for facial expression recognition. Specifically, we have designed the CoT mechanism from three key perspectives: key observations, overall emotional interpretation, and conclusion. The key observations describe the AU's name, intensity, and associated emotions. The overall emotional interpretation provides an analysis based on multiple AUs and their interactions, identifying the dominant emotions and their relationships. Finally, the conclusion presents the final expression label derived from the preceding analysis. Furthermore, we also introduce the Exp-CoT Engine, designed to construct this expression CoT and generate instruction-description data for training our ExpLLM. Extensive experiments on the RAF-DB and AffectNet datasets demonstrate that ExpLLM outperforms current state-of-the-art FER methods. ExpLLM also surpasses the latest GPT-4o in expression CoT generation, particularly in recognizing micro-expressions where GPT-4o frequently fails.

Introduction

Traditional Facial Expression Recognition (FER) systems often operate as "black boxes," identifying emotions like "sadness" or "happiness" without providing the underlying reasoning. While the Facial Action Coding System (FACS) can identify specific muscle movements known as Action Units (AUs), standard models typically output raw intensities without explaining how these movements interact to form a holistic emotional state. Furthermore, general-purpose multimodal models, such as GPT-4o, often struggle with "micro-expressions"—subtle, fleeting cues essential for accurate detection. ExpLLM addresses these gaps by introducing a "Chain of Thought" (CoT) framework that provides interpretable, step-by-step reasoning for facial analysis.

Methodology

The researchers developed a two-part solution to bridge the gap between raw facial data and emotional interpretation: the Exp-CoT Engine and the ExpLLM model.

The Exp-CoT Engine

The Exp-CoT Engine is a sophisticated data construction pipeline. It utilizes a specialized AU regression model to extract precise muscle movement data from images. This data is then processed by an LLM to generate a structured reasoning chain consisting of:

  • Key Observations: Detailed descriptions of specific AUs and their intensities.
  • Overall Emotional Interpretation: An analysis of how these individual movements relate to each other to form a dominant emotion.
  • Conclusion: The final, derived emotion label.

A critical feature of this engine is the Feedback Reflection loop. If the generated reasoning does not align with the ground-truth label, the engine provides corrective feedback to refine the reasoning until it is accurate.

The Exp-CoT Engine pipeline, illustrating the flow from AU regression to structured reasoning and feedback reflection.

ExpLLM Architecture

ExpLLM is the final model trained for deployment. It adopts a LLaVA-style architecture, combining a ViT-L/14 visual encoder with a Vicuna-7B large language model via a trainable projector. The model is fine-tuned using Low-Rank Adaptation (LoRA) on a new benchmark of approximately 50,000 high-quality image-CoT pairs. This architecture allows the model to look at a face and generate both the reasoning chain and the final label in a single pass.

The ExpLLM architecture, showing the integration of visual encoding and LLM reasoning via LoRA fine-tuning.

Evaluation and Results

Datasets

The study utilized benchmark FER datasets including RAF-DB (15,000 images) and AffectNet (subset of ~37,000 images). Additionally, the authors introduced a new benchmark of 49,824 image-CoT pairs to serve as a standard for interpretable FER. Cross-dataset evaluation was performed on ExpW to test generalization.

Key Findings

  • State-of-the-Art Accuracy: ExpLLM achieved 91.03% accuracy on RAF-DB, surpassing previous leading methods.
  • Superior Reasoning: Using a novel Exp-CoT Score (an LLM-based metric for logical consistency), ExpLLM scored 0.76, significantly higher than GPT-4o's 0.63.
  • Micro-expression Detection: Qualitative analysis demonstrated that ExpLLM consistently identifies subtle cues, such as slight lip corner pulls, which general-purpose models like GPT-4o often miss or hallucinate.
  • Reasoning as a Regularizer: Ablation studies indicated that the task of generating an explanation actually improves the model's classification accuracy, suggesting that reasoning acts as a powerful regularizer.

Conclusion and Future Work

ExpLLM demonstrates that grounding Large Language Models in precise facial geometry significantly enhances both the accuracy and interpretability of facial expression recognition. While the current model is limited to static images and carries a higher computational load than simple classifiers, it sets a new standard for transparent AI in affective computing. Future research will focus on extending this CoT framework to dynamic video sequences and optimizing the model for real-time edge applications.

Summary updated

Abstract

Facial expression recognition (FER) is a critical task in multimedia with significant implications across various domains. However, analyzing the causes of facial expressions is essential for accurately recognizing them. Current approaches, such as those based on facial action units (AUs), typically provide AU names and intensities but lack insight into the interactions and relationships between AUs and the overall expression. In this paper, we propose a novel method called ExpLLM, which leverages large language models to generate an accurate chain of thought (CoT) for facial expression recognition. Specifically, we have designed the CoT mechanism from three key perspectives: key observations, overall emotional interpretation, and conclusion. The key observations describe the AU's name, intensity, and associated emotions. The overall emotional interpretation provides an analysis based on multiple AUs and their interactions, identifying the dominant emotions and their relationships. Finally, the conclusion presents the final expression label derived from the preceding analysis. Furthermore, we also introduce the Exp-CoT Engine, designed to construct this expression CoT and generate instruction-description data for training our ExpLLM. Extensive experiments on the RAF-DB and AffectNet datasets demonstrate that ExpLLM outperforms current state-of-the-art FER methods. ExpLLM also surpasses the latest GPT-4o in expression CoT generation, particularly in recognizing micro-expressions where GPT-4o frequently fails.
BibTeX
@article{lan2025expllm,
  author = {Xing Lan and Jian Xue and Ji Qi and Dongmei Jiang and Ke Lu and Tat-Seng Chua},
  title = {{ExpLLM: Towards Chain of Thought for Facial Expression Recognition}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=1b7AJ5DiV9},
}
Figure