See Your Emotion from Gait Using Unlabeled Skeleton Data

Haifeng Lu, Xiping Hu, Bin HuAAAI 2023 · DOI

TL;DR

This paper introduces CAGE, a novel self-supervised learning framework for recognizing human emotions from skeleton-based gait data. CAGE addresses the limitations of existing methods by employing "ambiguity samples" and a cross-coordinate contrastive learning approach to learn robust gait representations without requiring labeled data, achieving performance comparable to or even exceeding supervised methods.

What Problem Does This Paper Address?

Recognizing human emotions is a crucial task in affective computing, with prior work leveraging physiological signals (like EEG) and non-physiological cues (facial expressions, tone of voice, walking styles). However, relying on self-reported emotions can be unreliable, and other cues might be influenced by environmental factors. Gait, observable from a distance and without subject cooperation, presents a promising avenue. While studies link gait to emotions, and 3D skeleton data is increasingly available, existing methods for gait-based emotion recognition often rely on handcrafted features or supervised deep learning approaches. These supervised methods demand extensive labeled datasets, which are costly and time-consuming to create. Furthermore, the inherent similarity between different emotional gaits can lead to annotation inconsistencies and mislabeling. This paper tackles the challenge of learning effective gait representations for emotion recognition from unlabeled skeleton data.

Methods

The core of CAGE is a self-supervised contrastive learning framework. It builds upon the idea of pushing gait sequences into a hypersphere space. The key innovations are:

  • Ambiguity Contrastive Learning (ACL): This component introduces "ambiguity samples" by transforming original gait sequences through changes in playback speed and applying shear transformations to joint positions. These transformed sequences, while derived from the same original gait, are designed to have ambiguous semantic information. By incorporating these ambiguity samples into the memory bank (alongside negative samples in some variants), the model learns to be more robust to semantic variations and improves generalization.
  • Cross-Coordinate Contrastive Learning (C³L): Recognizing that Cartesian coordinates capture joint positions and Spherical coordinates capture joint angles, CAGE leverages both. It converts the 3D skeleton data into both Cartesian and Spherical representations. Then, it applies contrastive learning to both streams simultaneously, using the Cartesian stream to guide the learning in the Spherical stream. This cross-coordinate supervision aims to capture richer, more consistent semantic information.
  • Memory Bank Strategies: CAGE explores three strategies for updating the memory bank:
  • Hybrid Samples Memory Bank (HSMB): Blends ambiguity samples directly into the memory bank as negative examples.
  • Ambiguity Samples Memory Bank (ASMB): Focuses on similarity between positive and ambiguity samples, removing the need for traditional negative samples.
  • Cooperation Samples Memory Bank (CSMB): Combines the base contrastive loss with a positive loss that increases similarity between ambiguity samples and positive samples.

The overall framework combines these components to learn gait representations.

Datasets

The paper evaluates CAGE on the Emotion-Gait (E-Gait) dataset. This dataset contains 2,177 gait samples across four emotions: happiness, sadness, neutral, and angry. It comprises two subsets: one where participants imagined emotions while walking, and another from the ELMD dataset annotated by 10 annotators. All samples are skeleton data with 16 body joints. The authors use a fixed 4:1 split for training and testing.

Key Insights and Ideas

  • Ambiguity is a Feature, Not a Bug: The paper's central insight is that introducing controlled "ambiguity" into gait sequences (via speed changes and shear transformations) can actually improve self-supervised learning. This ambiguity helps the model learn more robust and diverse gait representations, making it less sensitive to minor variations.
  • The Power of Multiple Perspectives: By combining Cartesian and Spherical coordinate representations, CAGE captures complementary information about gait. This cross-coordinate supervision is more effective than using either coordinate system alone, leading to better semantic consistency.
  • Self-Supervision Can Rival Supervision: The results strongly suggest that well-designed self-supervised methods, like CAGE, can achieve performance comparable to or even better than traditional supervised approaches, significantly reducing the reliance on expensive labeled data.
  • Gait Semantics are Sensitive to Speed: The study highlights how playback speed significantly impacts gait semantics, reinforcing the rationale behind using speed variations for ambiguity generation.

Key Results

  • CAGE achieves state-of-the-art performance among unsupervised methods, outperforming existing approaches by a significant margin (e.g., 2.76%-6.66% in linear evaluation).
  • When compared to supervised methods, CAGE demonstrates comparable or superior performance, particularly in linear and finetuned evaluation protocols. For instance, in finetuned evaluation, CAGE surpasses other methods by 3.21%-11.01%.
  • In semi-supervised settings, CAGE shows remarkable effectiveness, outperforming other models by a large margin (8.03%-21.10% accuracy improvement) even with only 5% of labeled data.
  • Ablation studies show that introducing ACL improves accuracy by 2.29%, and the combination of ACL and C³L (CAGE) yields the highest accuracy (79.59%).
  • Qualitative results using t-SNE visualization show that CAGE produces more discriminative clusters of emotion embeddings compared to the baseline GE-CLR.

Limitations and Future Work

  • Convergence Challenges with C³L: The paper notes that C³L, with its two input streams (Cartesian and Spherical), can be difficult to converge during pre-training, requiring an increased number of epochs to achieve stable gait representations.
  • ASMB Instability: While the ASMB strategy demonstrates the potential of learning without negative samples, its performance is noted as "not stable."
  • Future Work: The authors suggest that the framework's effectiveness in learning from limited labeled data opens avenues for more efficient semi-supervised emotion recognition systems. Further exploration into more sophisticated ambiguity generation techniques and cross-coordinate fusion strategies could also be beneficial.

Summary updated

BibTeX
@inproceedings{lu2023see,
  author = {Haifeng Lu and Xiping Hu and Bin Hu},
  title = {{See Your Emotion from Gait Using Unlabeled Skeleton Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25272},
}
Figure