Fusion in Context: A Multimodal Approach to Affective State Recognition

Youssef Mohamed, Séverin Lemaignan, Arzu Güneysu, Patric Jensfelt, Christian Smith

TL;DR

This paper presents a transformer-based model that significantly improves emotion recognition by treating situational context as a distinct data modality. By fusing facial action units and thermal data with rich textual descriptions of the environment and facial expressions (generated via GPT-4V), the researchers achieved an 89% F1 score in identifying states like frustration and enjoyment.

What Problem Does This Paper Address?

Recognizing human emotion is a "holy grail" for Human-Robot Interaction (HRI), but it is notoriously difficult because expressions are ambiguous. A smile can indicate genuine joy, or it can be a mask for sarcasm or discomfort depending on the situation.

Most current systems rely on "unimodal" data (just video or just heart rate) or "context-blind" multimodal fusion. These systems often fail because they don't account for the why behind an expression. The authors argue that to truly understand affect, a system must integrate physiological signals (thermal data), physical movements (action units), and the surrounding environmental context simultaneously.

Methods

The authors propose a hierarchical transformer-based architecture designed for "context-aware" recognition. The pipeline consists of three main stages:

  1. Modality-Specific Encoding: The model uses separate encoders for three distinct data streams:
  2. Visual: 16 Facial Action Units (AUs) extracted via OpenFace.
  3. Thermal: Temperature data from four facial Regions of Interest (ROIs) like the nose and forehead.
  4. Textual Context: This is the most novel part. They used GPT-4(V) to analyze video frames and generate brief sentences describing the user's facial dynamics. They then combined these with game-state info (e.g., "The person is playing Pacman on easy mode") and converted the text into 3072-dimensional vectors using OpenAI’s embedding-large model.
  5. Additive Fusion: Instead of complex attention-based merging at the start, they use a straightforward additive fusion to combine the representations from the three encoders.
  6. Shared Transformer Encoder: This fused representation is fed into a shared transformer to capture temporal dependencies—how the signals change over a 7-second window.

Datasets

The study utilized a dataset of participants playing a tangible tabletop Pacman game. The game was specifically designed with different difficulty levels and configurations to elicit four affective states:

  • Neutral (Baseline)
  • Enjoyment
  • Boredom
  • Frustration

The dataset included synchronized RGB video, thermal imaging, and game-play logs.

Key Insights and Ideas

  • Context as a First-Class Citizen: The paper demonstrates that context shouldn't just be a "weight" applied to other features; it works best when treated as its own modality with its own embedding space.
  • The "Noise" of Poor Context: Interestingly, adding "Game-Only Context" (just the difficulty level) actually decreased performance compared to using no context at all. This suggests that low-quality or overly simplistic contextual data acts as noise that confuses the model.
  • LMMs as Feature Extractors: Rather than using GPT-4V to predict the emotion directly (which only yielded a 23% F1 score), they used it to provide high-level semantic descriptions. This "human-like" description of a face proved to be a powerful feature for the downstream transformer model.
  • Thermal vs. Visual: Thermal data captures physiological changes (like a cold nose during stress) that are harder to "fake" than muscle movements (Action Units), making the combination highly robust.

Key Results

The inclusion of "Full Context" (game state + facial descriptions) led to the highest performance:

  • Best F1 Score: 89% (using Thermal + AU + Full Context).
  • Ablation Comparison:
    • Thermal + AU (No context): 84%
    • Action Units Only: 65%
    • Thermal Data Only: 30%
    • GPT-4(V) Zero-shot Prediction: 23%
  • State-Specific Accuracy: The model was nearly perfect at detecting Enjoyment (96.9%) and very strong at Neutral (91.1%). Boredom remained the most challenging state to classify at 78.3%, often being confused with the neutral baseline.

Limitations and Future Work

  • Computational Latency: While the authors designed the system for real-time HRI, the reliance on external APIs (OpenAI's GPT-4V and embeddings) introduces latency and privacy concerns that might be problematic for edge-deployed robots.
  • Context Complexity: The "Full Context" used in this study is still relatively constrained to a game environment. Future work is needed to see how this approach scales to "in-the-wild" social scenarios where the context is much more fluid and less structured than a Pacman game.
  • Temporal Window: The study uses a fixed 7-second window. The authors note that different affective states manifest over different timescales, suggesting that a multi-scale temporal approach might further improve accuracy.

Summary updated

BibTeX
@inproceedings{mohamed2025fusion,
  author = {Youssef Mohamed and Séverin Lemaignan and Arzu Güneysu and Patric Jensfelt and Christian Smith},
  title = {{Fusion in Context: A Multimodal Approach to Affective State Recognition}},
  booktitle = {RO-MAN},
  year = {2025},
  doi = {10.1109/ro-man63969.2025.11217904},
}
Figure