Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values

Vaibhav Mehra, Guy Laban, Hatice GunesarXiv 2025 · DOI · arXiv

Abstract

Traditional emotion recognition in AI relies on Vision-Language Models (VLMs) processing raw images, which is computationally intensive and raises privacy concerns. This paper investigates an alternative: can Large Language Models (LLMs) interpret human facial expressions using only structured numerical data—specifically, Valence and Arousal (VA) values? Our findings indicate that while LLMs struggle with rigid categorization into discrete emotion labels, they excel at generating nuanced, human-like semantic descriptions. This suggests a lightweight, privacy-preserving pathway for integrating emotional intelligence into AI systems.

Introduction

As AI agents become increasingly integrated into social environments, the ability to perceive and respond to human emotions is paramount. Currently, this "affective computing" is dominated by VLMs that analyze raw facial imagery. However, this approach is often a "black box" that demands significant resources and poses substantial privacy risks. It is often unclear whether these models truly understand emotional states or are simply performing linguistic guesswork based on visual context.

This study explores a more efficient and transparent alternative: extracting the core dimensions of emotion—Valence (the positivity or negativity of an emotion) and Arousal (the intensity or activation level)—and feeding these numerical values into text-based LLMs. By bypassing raw visual input, we test whether LLMs can infer affective meaning from abstract, structured representations.

Methodology

The researchers employed the FaceChannel tool to extract VA values from facial images. These numerical representations were then presented to several LLMs—GPT-4o, GPT-4o-mini, and LLaMA 3.2 8B—through text-based prompts. The models were evaluated on two primary tasks:

  1. Categorization Task: Mapping VA values to discrete emotion labels. This included a "basic" task (7 Ekman emotions) and a "complex" task (26 nuanced states such as yearning or annoyance).
  2. Semantic Description Task: Generating open-ended text descriptions of a person's mental state based solely on VA values.

To assess the quality of the semantic descriptions, the authors utilized text embedding models (Word2Vec, BERT, and Transformers) to measure the cosine similarity between LLM-generated text and ground-truth human annotations.

Datasets

The study utilized two key datasets:

  • IIMI Dataset: Comprising 700 images of Indian individuals expressing basic emotions, used for the basic categorization task.
  • Emotic Dataset: Featuring 3,047 images in diverse social scenarios, annotated with 26 complex emotion categories, human-labeled VA values, and descriptive text. This dataset supported the complex categorization and semantic description tasks.

Results and Discussion

The Failure of Rigid Categorization

The results for the categorization task were notably poor. For basic emotions, GPT-4o-mini achieved an overall accuracy of only ~30.4%. The model exhibited significant bias, frequently defaulting to "Happy" or "Sad" while failing to recognize "Fear" or "Anger" from numerical data alone.

Table 1: Comparison of categorization performance between GPT-4o-mini and FaceChannel across basic emotion categories.

In the complex categorization task, GPT-4o-mini correctly identified at least one relevant category for 49.67% of images but achieved exact matches in only 18.32% of cases. Interestingly, the smaller GPT-4o-mini model outperformed the larger GPT-4o, suggesting that increased parameter count does not necessarily improve numerical-to-affective reasoning.

The Success of Semantic Nuance

While LLMs struggled to fit emotions into "rigid boxes," they demonstrated remarkable proficiency in the semantic description task. By generating free-form text, the models captured the "vibe" and subtle gradients of human emotion that discrete labels often miss.

Table 2: Qualitative comparison of human descriptions versus GPT-4o-mini descriptions based on Valence and Arousal values.

Quantitative analysis confirmed this alignment. GPT-4o-mini's descriptions achieved high cosine similarity scores (0.79 with BERT and 0.81 with Word2Vec), indicating that the generated text was highly representative of how humans naturally describe the same facial expressions. This supports the psychological theory that emotion is a continuous gradient rather than a set of discrete categories.

Conclusion and Future Directions

This research demonstrates that LLMs can effectively interpret emotional states from low-dimensional numerical data, provided they are allowed to use their natural linguistic strengths. While VA values alone may lack full situational context, they offer a privacy-friendly and computationally efficient alternative to raw image processing.

Future work should explore hybrid models that combine VA values with other lightweight modalities, such as speech prosody or facial action units. Additionally, direct benchmarking against state-of-the-art VLMs will help clarify the trade-offs between accuracy, cost, and user privacy in the next generation of emotionally intelligent AI.

Summary updated

Abstract

Large Language Models primarily operate through text-based inputs and outputs, yet human emotion is communicated through both verbal and non-verbal cues, including facial expressions. While Vision-Language Models analyze facial expressions from images, they are resource-intensive and may depend more on linguistic priors than visual understanding. To address this, this study investigates whether LLMs can infer affective meaning from dimensions of facial expressions-Valence and Arousal values, structured numerical representations, rather than using raw visual input. VA values were extracted using Facechannel from images of facial expressions and provided to LLMs in two tasks: (1) categorizing facial expressions into basic (on the IIMI dataset) and complex emotions (on the Emotic dataset) and (2) generating semantic descriptions of facial expressions (on the Emotic dataset). Results from the categorization task indicate that LLMs struggle to classify VA values into discrete emotion categories, particularly for emotions beyond basic polarities (e.g., happiness, sadness). However, in the semantic description task, LLMs produced textual descriptions that align closely with human-generated interpretations, demonstrating a stronger capacity for free text affective inference of facial expressions.
BibTeX
@misc{mehra2025beyond,
  author = {Vaibhav Mehra and Guy Laban and Hatice Gunes},
  title = {{Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values}},
  year = {2025},
  eprint = {2502.06875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.06875},
}
Figure