Abstract
Recognizing human emotions through gait (walking patterns) offers a robust alternative to facial recognition, particularly when subjects are masked or viewed from a distance. However, existing methods often fail to capture the complex, long-range dependencies and the complementary relationships between raw joint movements and affective features. We present TNTC (Two-stream Network with Transformer-based Complementarity), a novel framework that encodes skeleton sequences into images and utilizes a Transformer-based module to fuse joint and affective data streams. TNTC achieves a state-of-the-art accuracy of 85.97% on the Emotion-Gait dataset, significantly outperforming existing graph-based and sequence-based models.
Introduction
While facial expressions are the primary modality for emotion recognition, they are easily obscured or manipulated. Gait provides a non-intrusive, difficult-to-mask biometric signal. Current research in gait-based emotion recognition typically employs sequence-based (LSTM), graph-based (GCN), or image-based (CNN) approaches. Despite their successes, these methods suffer from two primary limitations: rigid feature fusion, which fails to exploit the synergy between different feature types, and an inability to model long-range spatial-temporal dependencies (e.g., the relationship between a slumped shoulder at the start of a stride and a slow foot-fall later).
Methodology
1. Skeleton Sequence Encoding
TNTC treats gait recognition as a multi-stream image classification task. The raw skeleton data is encoded into two distinct 2D representations:
- Skeleton Joint Image (SJI): This encodes the 3D coordinates of joints over time into a temporal image.
- Affective Feature Image (AFI): This encodes projection angles of joints across three geometric planes. These angles provide viewpoint-invariant features, ensuring the model remains robust regardless of camera placement.
2. Two-Stream Architecture and TCM
The core architecture consists of two parallel ResNet-34 backbones. To bridge these streams, the authors introduce the Transformer-Based Complementarity Module (TCM). Inserted after each residual block, the TCM uses self-attention to identify and fuse complementary features between the "Joint" and "Affective" streams. By applying this fusion hierarchically across four levels, the network learns to correlate low-level joint jitters with high-level postural shifts.
Experimental Results
Performance Comparison
The model was evaluated on the Emotion-Gait dataset, comprising 2,177 real-world sequences labeled with four emotions: Happy, Sad, Angry, and Neutral. TNTC established a new state-of-the-art benchmark, achieving 85.97% accuracy.
Ablation Studies
Ablation experiments demonstrated the critical role of the TCM. Removing these modules caused accuracy to drop to 81.53%, confirming that the Transformer-based fusion is the primary driver of the model's superior performance. Visualizations of attention maps further confirmed that the TCM focuses specifically on the "cross-regions" where the two data streams provide the most synergistic information.
Conclusion
TNTC demonstrates that treating gait as a multi-stream image classification problem, combined with attention-based feature fusion, effectively captures the nuances of human emotion. While the current study focuses on four primary emotions, future work will explore more complex affective states and the integration of additional dynamic features like velocity and acceleration to further refine recognition accuracy.
Summary updated