Human-Centric Emotion Estimation Based on Correlation Maximization Considering Changes With Time in Visual Attention and Brain Activity

Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

Abstract

Understanding human emotion is a cornerstone of affective computing, yet current systems often struggle to balance accuracy with user comfort. This paper introduces a novel human-centric emotion estimation framework that integrates temporal visual attention patterns with brain activity. By employing a 4th-order Gaze and Image Tensor (GIT) and Canonical Correlation Analysis (CCA), the system "trains" eye-gaze features using functional Near-Infrared Spectroscopy (fNIRS) data. This allows for highly accurate emotion prediction during inference using only non-intrusive eye-tracking hardware. Evaluated on the Art Photo Dataset, the proposed method achieved a superior F1-measure of 0.77, outperforming both single-modality models and complex deep learning architectures.

Introduction

Affective computing aims to bridge the gap between machines and human "sensibility." While traditional methods extract features from static images or text, they often fail to capture how an individual truly feels. Multimodal approaches combining eye-gaze and brain signals (e.g., EEG, fNIRS) offer high accuracy but are hindered by significant user burden—wearing brain-sensing caps is impractical for daily use. Furthermore, many existing models treat visual stimuli as static, ignoring the temporal sequence of how a user's attention shifts across an image.

This research addresses these challenges by proposing a system that captures the "story" of visual attention over time and uses brain activity only during the learning phase to refine gaze-based feature interpretation.

Methodology

The proposed pipeline utilizes high-order tensors and correlation maximization to align visual attention with internal emotional states.

Overview of the proposed emotion estimation pipeline, including feature extraction, CCA-based alignment, and ELM classification.

Gaze and Image Tensor (GIT) Construction

Rather than treating an image as a flat input, the researchers construct a 4th-order tensor. The dimensions include pixel location (x, y), color channels, and, crucially, time. This structure allows the model to track frame-by-frame shifts in user attention, recognizing that the initial focus often carries the strongest emotional weight.

Construction of the Gaze and Image Tensor (GIT) capturing temporal changes in visual attention.

Feature Extraction and Alignment

Deep visual features are extracted from gazed-at areas using transfer learning with state-of-the-art CNNs (Xception, InceptionResNet-v2, and DenseNet201). The core innovation lies in the Brain-Gaze Alignment via Canonical Correlation Analysis (CCA). During the training phase, fNIRS data is recorded alongside gaze data. CCA finds a mathematical projection that maximizes the correlation between these two modalities. This process creates a "Brain activity-based Gaze and Image Tensor" (BGIT), effectively embedding brain-level insights into the gaze features.

Classification

To handle the resulting high-dimensional data, the system employs Generalized Tensor Discriminant Analysis (GTDA) for dimensionality reduction. The final emotion classification is performed by an Extreme Learning Machine (ELM), chosen for its efficiency and speed with relatively small datasets.

Experimental Setup

The system was validated using the Art Photo Dataset, which contains images across eight emotional categories (e.g., Amusement, Fear, Sadness). Ten participants viewed 80 images for 10 seconds each while their gaze was tracked by a Tobii Eye Tracker 4C and brain activity was monitored via SHIMADZU LIGHTNIRS.

Results and Discussion

The proposed method demonstrated significant improvements over existing techniques:

  • High Accuracy: The integrated model achieved an average F1-measure of 0.77.
  • Modality Comparison: It substantially outperformed single-modality models (gaze-only or brain-only), which typically ranged between 0.53 and 0.60.
  • Efficiency: The tensor-based approach proved more effective than complex architectures like Deep CCA (0.40) and LSTMs (0.53), likely due to its superior handling of spatio-temporal structures in limited-sample environments.

A key insight is the "Brain-Free" inference capability. Once the CCA projection matrix is learned, the fNIRS cap is no longer required. The system can estimate emotions using only the eye tracker, as it has already learned how specific gaze patterns correlate with brain-indicated emotional states.

Conclusion

This research provides a practical path toward non-intrusive, human-centric emotion estimation. By accounting for the temporal dynamics of attention and individual sensibility, the model achieves high performance without the ongoing need for cumbersome brain-sensing hardware. Future work will focus on expanding the participant pool to improve generalizability and developing cross-user transfer learning to reduce individual training requirements.

Summary updated

Abstract

A human-centric emotion estimation method based on correlation maximization with consideration of changes with time in visual attention and brain activity when viewing images is proposed in this paper. Owing to the recent developments of many kinds of biological sensors, many researchers have focused on multimodal emotion estimation using both eye gaze data and brain activity data for improving the quality of emotion estimation. In this paper, a novel method that focuses on the following two points is introduced. First, in order to reduce the burden on users, we obtain brain activity data from users only in the training phase by using a projection matrix calculated by canonical correlation analysis (CCA) between gaze-based visual features and brain activity-based features. Next, for considering the changes with time in both visual attention and brain activity, we obtain novel features based on CCA-based projection in each time unit. In order to include these two points, the proposed method analyzes a fourth-order gaze and image tensor for which modes are pixel location, color channel and the changes with time in visual attention. Moreover, in each time unit, the proposed method performs CCA between gaze-based visual features and brain activity-based features to realize human-centric emotion estimation with a high level of accuracy. Experimental results show that accurate human emotion estimation is achieved by using our new human-centric image representation.
BibTeX
@article{moroto2020human,
  author = {Yuya Moroto and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Human-Centric Emotion Estimation Based on Correlation Maximization Considering Changes With Time in Visual Attention and Brain Activity}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3036908},
}
Figure