Abstract
Traditional emotion recognition in AI relies on Vision-Language Models (VLMs) processing raw images, which is computationally intensive and raises privacy concerns. This paper investigates an alternative: can Large Language Models (LLMs) interpret human facial expressions using only structured numerical data—specifically, Valence and Arousal (VA) values? Our findings indicate that while LLMs struggle with rigid categorization into discrete emotion labels, they excel at generating nuanced, human-like semantic descriptions. This suggests a lightweight, privacy-preserving pathway for integrating emotional intelligence into AI systems.
Introduction
As AI agents become increasingly integrated into social environments, the ability to perceive and respond to human emotions is paramount. Currently, this "affective computing" is dominated by VLMs that analyze raw facial imagery. However, this approach is often a "black box" that demands significant resources and poses substantial privacy risks. It is often unclear whether these models truly understand emotional states or are simply performing linguistic guesswork based on visual context.
This study explores a more efficient and transparent alternative: extracting the core dimensions of emotion—Valence (the positivity or negativity of an emotion) and Arousal (the intensity or activation level)—and feeding these numerical values into text-based LLMs. By bypassing raw visual input, we test whether LLMs can infer affective meaning from abstract, structured representations.
Methodology
The researchers employed the FaceChannel tool to extract VA values from facial images. These numerical representations were then presented to several LLMs—GPT-4o, GPT-4o-mini, and LLaMA 3.2 8B—through text-based prompts. The models were evaluated on two primary tasks:
- Categorization Task: Mapping VA values to discrete emotion labels. This included a "basic" task (7 Ekman emotions) and a "complex" task (26 nuanced states such as yearning or annoyance).
- Semantic Description Task: Generating open-ended text descriptions of a person's mental state based solely on VA values.
To assess the quality of the semantic descriptions, the authors utilized text embedding models (Word2Vec, BERT, and Transformers) to measure the cosine similarity between LLM-generated text and ground-truth human annotations.
Datasets
The study utilized two key datasets:
- IIMI Dataset: Comprising 700 images of Indian individuals expressing basic emotions, used for the basic categorization task.
- Emotic Dataset: Featuring 3,047 images in diverse social scenarios, annotated with 26 complex emotion categories, human-labeled VA values, and descriptive text. This dataset supported the complex categorization and semantic description tasks.
Results and Discussion
The Failure of Rigid Categorization
The results for the categorization task were notably poor. For basic emotions, GPT-4o-mini achieved an overall accuracy of only ~30.4%. The model exhibited significant bias, frequently defaulting to "Happy" or "Sad" while failing to recognize "Fear" or "Anger" from numerical data alone.
In the complex categorization task, GPT-4o-mini correctly identified at least one relevant category for 49.67% of images but achieved exact matches in only 18.32% of cases. Interestingly, the smaller GPT-4o-mini model outperformed the larger GPT-4o, suggesting that increased parameter count does not necessarily improve numerical-to-affective reasoning.
The Success of Semantic Nuance
While LLMs struggled to fit emotions into "rigid boxes," they demonstrated remarkable proficiency in the semantic description task. By generating free-form text, the models captured the "vibe" and subtle gradients of human emotion that discrete labels often miss.
Quantitative analysis confirmed this alignment. GPT-4o-mini's descriptions achieved high cosine similarity scores (0.79 with BERT and 0.81 with Word2Vec), indicating that the generated text was highly representative of how humans naturally describe the same facial expressions. This supports the psychological theory that emotion is a continuous gradient rather than a set of discrete categories.
Conclusion and Future Directions
This research demonstrates that LLMs can effectively interpret emotional states from low-dimensional numerical data, provided they are allowed to use their natural linguistic strengths. While VA values alone may lack full situational context, they offer a privacy-friendly and computationally efficient alternative to raw image processing.
Future work should explore hybrid models that combine VA values with other lightweight modalities, such as speech prosody or facial action units. Additionally, direct benchmarking against state-of-the-art VLMs will help clarify the trade-offs between accuracy, cost, and user privacy in the next generation of emotionally intelligent AI.
Summary updated