TNTC: two-stream network with transformer-based complementarity for gait-based emotion recognition

Chuanfei Hu, Weijie Sheng, Bo Dong, Xinde Li

Abstract

Recognizing human emotions through gait (walking patterns) offers a robust alternative to facial recognition, particularly when subjects are masked or viewed from a distance. However, existing methods often fail to capture the complex, long-range dependencies and the complementary relationships between raw joint movements and affective features. We present TNTC (Two-stream Network with Transformer-based Complementarity), a novel framework that encodes skeleton sequences into images and utilizes a Transformer-based module to fuse joint and affective data streams. TNTC achieves a state-of-the-art accuracy of 85.97% on the Emotion-Gait dataset, significantly outperforming existing graph-based and sequence-based models.

Introduction

While facial expressions are the primary modality for emotion recognition, they are easily obscured or manipulated. Gait provides a non-intrusive, difficult-to-mask biometric signal. Current research in gait-based emotion recognition typically employs sequence-based (LSTM), graph-based (GCN), or image-based (CNN) approaches. Despite their successes, these methods suffer from two primary limitations: rigid feature fusion, which fails to exploit the synergy between different feature types, and an inability to model long-range spatial-temporal dependencies (e.g., the relationship between a slumped shoulder at the start of a stride and a slow foot-fall later).

Methodology

1. Skeleton Sequence Encoding

TNTC treats gait recognition as a multi-stream image classification task. The raw skeleton data is encoded into two distinct 2D representations:

  • Skeleton Joint Image (SJI): This encodes the 3D coordinates of joints over time into a temporal image.
  • Affective Feature Image (AFI): This encodes projection angles of joints across three geometric planes. These angles provide viewpoint-invariant features, ensuring the model remains robust regardless of camera placement.
The TNTC architecture overview, showing the encoding of gait sequences into SJI and AFI, followed by the two-stream perception network.

2. Two-Stream Architecture and TCM

The core architecture consists of two parallel ResNet-34 backbones. To bridge these streams, the authors introduce the Transformer-Based Complementarity Module (TCM). Inserted after each residual block, the TCM uses self-attention to identify and fuse complementary features between the "Joint" and "Affective" streams. By applying this fusion hierarchically across four levels, the network learns to correlate low-level joint jitters with high-level postural shifts.

Experimental Results

Performance Comparison

The model was evaluated on the Emotion-Gait dataset, comprising 2,177 real-world sequences labeled with four emotions: Happy, Sad, Angry, and Neutral. TNTC established a new state-of-the-art benchmark, achieving 85.97% accuracy.

Comparison of TNTC against existing graph-based, sequence-based, and image-based methods on the Emotion-Gait dataset.

Ablation Studies

Ablation experiments demonstrated the critical role of the TCM. Removing these modules caused accuracy to drop to 81.53%, confirming that the Transformer-based fusion is the primary driver of the model's superior performance. Visualizations of attention maps further confirmed that the TCM focuses specifically on the "cross-regions" where the two data streams provide the most synergistic information.

Conclusion

TNTC demonstrates that treating gait as a multi-stream image classification problem, combined with attention-based feature fusion, effectively captures the nuances of human emotion. While the current study focuses on four primary emotions, future work will explore more complex affective states and the integration of additional dynamic features like velocity and acceleration to further refine recognition accuracy.

Summary updated

Abstract

Recognizing the human emotion automatically from visual characteristics plays a vital role in many intelligent applications. Recently, gait-based emotion recognition, especially gait skeletons-based characteristic, has attracted much attention, while many available methods have been proposed gradually. The popular pipeline is to first extract affective features from joint skeletons, and then aggregate the skeleton joint and affective features as the feature vector for classifying the emotion. However, the aggregation procedure of these emerged methods might be rigid, resulting in insufficiently exploiting the complementary relationship between skeleton joint and affective features. Meanwhile, the long range dependencies in both spatial and temporal domains of the gait sequence are scarcely considered. To address these issues, we propose a novel two-stream network with transformer-based complementarity, termed as TNTC. Skeleton joint and affective features are encoded into two individual images as the inputs of two streams, respectively. A new transformer-based complementarity module (TCM) is proposed to bridge the complementarity between two streams hierarchically via capturing long range dependencies. Experimental results demonstrate TNTC outperforms state-of-the-art methods on the latest dataset in terms of accuracy.
BibTeX
@inproceedings{hu2021tntc,
  author = {Chuanfei Hu and Weijie Sheng and Bo Dong and Xinde Li},
  title = {{TNTC: two-stream network with transformer-based complementarity for gait-based emotion recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp43922.2022.9746047},
}
Figure