iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis

Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, Guoying Zhao

Abstract

Traditional emotion recognition systems heavily rely on facial expressions, raising significant privacy concerns and susceptibility to masking. This paper introduces iMiGUE, a large-scale, identity-free video dataset focused on micro-gestures (MGs)—spontaneous, unintentional body movements that reveal internal emotional states. By masking facial and vocal data, the researchers demonstrate that body language alone can effectively classify emotional peaks. We further propose an Unsupervised Sequential Variational Autoencoder (U-S-VAE) to address the inherent class imbalance in micro-gesture data.

Introduction

In the field of Affective Computing, the face has traditionally been the primary modality for emotion analysis. However, facial data is highly sensitive biometric information, and individuals can often consciously mask their true feelings. Micro-gestures (MGs), such as rubbing hands or adjusting a collar, offer a more reliable "leakage" of subconscious emotional states. Unlike intentional gestures (e.g., waving), MGs are brief, subtle, and often occur "in the wild."

The primary challenge in MG research has been the lack of large-scale, publicly available datasets. Furthermore, existing datasets often focus on intentional actions rather than spontaneous emotional expressions. The iMiGUE dataset addresses these gaps while prioritizing privacy through identity-free processing.

The iMiGUE Dataset

The iMiGUE dataset consists of 359 videos (over 2,000 minutes) of post-match tennis press conferences from Grand Slam tournaments. This setting provides an objective ground truth for emotional states: winning or losing a match serves as a proxy for positive or negative emotional peaks, respectively.

The dataset features 72 players from 28 countries and contains 18,499 MG samples categorized into 32 classes. These classes are organized into a taxonomy covering the body, head, hands, and combinations thereof. A significant characteristic of iMiGUE is its "long-tail" distribution, where a few common gestures (like "shaking shoulders") appear much more frequently than others (like "biting nails").

Taxonomy of 32 micro-gesture categories and their frequency distribution in the iMiGUE dataset.

Methodology: Unsupervised Learning for Micro-Gestures

To handle the diversity and imbalance of the dataset, the authors developed the Unsupervised Sequential Variational Autoencoder (U-S-VAE). This model learns to extract meaningful features from body pose sequences without requiring manual labels for every sample.

The architecture consists of:

  1. Encoder: A multi-layer Bidirectional Long Short-Term Memory (BLSTM) network that processes the sequence of body key-points to capture temporal dynamics.
  2. Decoder: A standard LSTM network that attempts to reconstruct the original input sequence from a compressed latent representation ($Z_T$).

By training the model to minimize reconstruction error and KL-divergence loss, it learns a robust representation of movement patterns. These learned features are then used with a K-Nearest Neighbors (KNN) classifier for gesture recognition.

Architecture of the proposed Unsupervised Sequential Variational Autoencoder (U-S-VAE).

Experimental Results

The researchers benchmarked several state-of-the-art supervised and unsupervised models on iMiGUE. Supervised models like the Temporal Shift Module (TSM) achieved the highest Top-1 accuracy at 61.10%. However, the proposed unsupervised U-S-VAE reached 32.43%, outperforming previous unsupervised methods and demonstrating better resilience to the dataset's imbalanced nature.

In the final task of emotion prediction (classifying a video as "win" or "loss"), the model achieved 60% accuracy using recognized micro-gestures. Notably, this outperformed end-to-end models that attempted to predict emotion directly from raw video frames, confirming that micro-gestures are a potent intermediary for understanding human affect.

Conclusion

The iMiGUE dataset and the U-S-VAE framework represent a significant step toward privacy-preserving, identity-free emotion recognition. By shifting focus from the face to subtle body movements, this research provides a new pathway for building Emotional AI that is both more secure and harder to deceive. Future work will focus on refining the model's ability to distinguish between visually similar gestures and expanding the emotional labels beyond binary states.

Summary updated

BibTeX
@inproceedings{liu2021imigue,
  author = {Xin Liu and Henglin Shi and Haoyu Chen and Zitong Yu and Xiaobai Li and Guoying Zhao},
  title = {{iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01049},
}
Figure