Abstract
Recognizing human affective states remains a significant challenge in Human-Robot Interaction (HRI) due to the inherent ambiguity of facial expressions. This paper presents a novel transformer-based architecture that treats situational context as a distinct modality alongside visual and physiological data. By fusing facial action units (AUs) and thermal imaging with rich textual descriptions generated by Large Multimodal Models (LMMs), the proposed system achieves an 89% F1 score in classifying states such as enjoyment, boredom, and frustration. Our findings demonstrate that high-level semantic context significantly enhances the robustness of emotion recognition systems.
Introduction
The ability to accurately interpret human emotion is a cornerstone of effective Human-Robot Interaction. However, traditional unimodal or context-blind multimodal systems often struggle with the "ambiguity problem"—where a single physical expression, like a smile, can represent vastly different internal states depending on the environment. To address this, we propose a framework that integrates physiological signals (thermal data), physical movements (facial action units), and environmental context into a unified representation.
Methodology
The core of our approach is a hierarchical transformer-based pipeline designed for context-aware recognition. The architecture processes three distinct data streams:
- Visual Modality: 16 Facial Action Units (AUs) are extracted from RGB video using OpenFace to capture muscle movements.
- Thermal Modality: Temperature fluctuations are recorded from four facial Regions of Interest (ROIs)—the forehead, nose, cheeks, and lower lip—to capture physiological stress responses.
- Textual Context: This novel modality leverages GPT-4(V) to generate semantic descriptions of the user's facial dynamics. These are combined with game-state information (e.g., difficulty level) and embedded into a high-dimensional vector space.
These representations are merged via additive fusion and processed by a shared transformer encoder to capture temporal dependencies over a 7-second window.
Experimental Setup
The model was evaluated using a dataset of participants playing a tangible tabletop Pacman game. The game environment was manipulated with varying difficulty levels to elicit four target affective states: Neutral, Enjoyment, Boredom, and Frustration. Data collection included synchronized RGB video, thermal imaging, and game-play logs.
Results and Discussion
The integration of "Full Context" (game state plus facial descriptions) yielded the highest performance, achieving an F1 score of 89%. This represents a significant improvement over models using only physical features (84% F1) or unimodal data (65% for AUs, 30% for Thermal).
As shown in the confusion matrix, the model is exceptionally accurate at identifying Enjoyment (96.9%) and Neutral (91.1%) states. Boredom (78.3%) remains the most difficult to distinguish, often being misclassified as neutral. Interestingly, the study found that "Game-Only Context" (difficulty level without facial description) actually degraded performance, suggesting that simplistic context can act as noise.
Conclusion
Treating context as a "first-class citizen" in multimodal fusion significantly improves affective state recognition. While the use of LMMs as feature extractors provides powerful semantic insights, future work must address the computational latency of API-dependent models to enable seamless real-time HRI. Additionally, expanding this approach to unstructured "in-the-wild" social scenarios remains a key frontier for context-aware affective computing.
Summary updated