Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning

Mahsa Ehsanpour, Ian Reid, Hamid Rezatofighi

TL;DR

Social-MAE is a self-supervised framework that learns rich representations of multi-person motion by reconstructing masked joint trajectories in the frequency domain. By pre-training on unlabeled motion data, it achieves state-of-the-art performance across diverse high-level social tasks—such as pose forecasting, social grouping, and action detection—while demonstrating superior data efficiency.

What Problem Does This Paper Address?

While computer vision has made massive strides in detecting and tracking individuals, truly "understanding" a scene requires moving into the social realm. We need models that can predict where people will move next, identify who is interacting with whom (social grouping), and recognize complex social actions like "listening to someone."

The primary bottleneck for these high-level tasks is a lack of annotated data. Labeling social interactions is significantly more labor-intensive and subjective than drawing bounding boxes. Consequently, models trained purely via supervised learning often struggle to generalize to new environments. Social-MAE addresses this by leveraging self-supervised learning, allowing models to learn the underlying "grammar" of human motion and interaction from vast amounts of unlabeled pose data before being fine-tuned for specific tasks.

Methods

The authors introduce an asymmetric Masked Autoencoder (MAE) architecture specifically tailored for spatio-temporal motion data.

  1. Input & Frequency Domain: Instead of feeding raw Cartesian coordinates $(x, y, z)$ directly into the model, the joint trajectories are transformed into the frequency domain using a Discrete Cosine Transform (DCT). This captures the periodic and continuous nature of human movement more effectively than raw sequences.
  2. Tube Masking: To prevent the model from "cheating" by interpolating between adjacent frames, the authors use a strategy called "Tube Masking." They randomly select a subset of joints and mask their entire trajectory across the time window.
  3. Asymmetric Architecture: The framework consists of a deep Transformer encoder and a shallow, lightweight Transformer decoder. The encoder only processes the visible (unmasked) joint trajectories, making it highly efficient. The decoder then attempts to reconstruct the full set of trajectories in the DCT space.
  4. Downstream Adaptation: Once pre-trained, the decoder is discarded. The pre-trained encoder is then coupled with task-specific decoders for three distinct social tasks:
  5. Social Pose Forecasting: Predicting future joint positions.
  6. Social Grouping: Determining if individuals belong to the same functional group.
  7. Social Action Understanding: Recognizing individual and interaction-based actions.
Fig. 1: On the left, the initial tokens to the social-MAE are joints’ trajectory of all people in the scene. We randomly mask a subset of input tokens. The encoder performs on a small subset of input tokens (unmasked tokens only). The small decoder is then applied on the latent representation of unmasked tokens as well as the masked ones to reconstruct the original input joints’ trajectory. On the right and after pre-training, the decoder is replaced with a task-specific decoder for each downstream task and the pre-trained encoder is applied on the full-set of multi-person joints’ trajectory.

Datasets

The model was evaluated across four major datasets covering both 2D and 3D human poses:

  • 3D Motion Forecasting: Evaluated on 3DPW, AMASS, CMU-Mocap, and MuPoTS-3D. The authors even synthesized multi-person scenes by mixing single-person sequences from AMASS.
  • Social Interaction: Evaluated on JRDB-Act, a large-scale dataset captured by a mobile robot in crowded environments, containing over 2.8 million social action labels.

Key Insights and Ideas

  • The Power of Sparsity: Similar to findings in the image and video domains, the authors found that masking a significant portion of the data (50%) forces the encoder to learn high-level structural and social dependencies rather than just local correlations.
  • Motion is Redundant: The optimal masking ratio for motion (50%) is lower than for video (90%). This suggests that while motion data is redundant, it is less so than dense pixel grids, requiring a more careful balance to ensure the reconstruction task remains challenging but solvable.
  • Social Context Matters: By processing all individuals in a scene simultaneously within the transformer, the encoder naturally learns to represent how one person's motion influences another's—a critical requirement for social understanding.

Key Results

Social-MAE sets a new state-of-the-art (SOTA) across all tested tasks:

  • Pose Forecasting: On the SoMoF benchmark, Social-MAE achieved an overall VIM (Visibility-Ignored Metric) of 48.6, outperforming the previous SOTA (49.4).
  • Social Grouping: On the JRDB-Act test set, it achieved a mean Average Precision (mAP) of 32.4, significantly higher than previous methods that relied on visual data or complex hand-crafted rules.
  • Action Detection: It outperformed existing models on JRDB-Act, reaching an mAP of 11.2 on the test set, proving that motion trajectories alone can be more informative for social actions than raw video frames in some contexts.
  • Data Efficiency: Ablation studies showed that the pre-trained model is much more resilient to reductions in labeled training data compared to models trained from scratch.

Limitations and Future Work

The authors identify a few key areas for improvement:

  • Visual Ambiguity: The model occasionally produces false positives for actions like "talking to someone" when people are simply moving in close proximity. The authors suggest that incorporating visual features (pixels) alongside pose data could help resolve these social nuances.
  • 2D vs. 3D Grouping: Currently, grouping failures often occur because the model relies on 2D trajectories, which can be deceptive in crowded scenes. Moving toward 3D data for social grouping is a proposed future direction.
  • Identity Sensitivity: While using pose data minimizes the risk of exposing sensitive identity details compared to video, the authors acknowledge that behavioral patterns themselves can be sensitive.

Summary updated

Abstract

For a complete comprehension of multi-person scenes, it is essential to go beyond basic tasks like detection and tracking. Higher-level tasks, such as understanding the interactions and social activities among individuals, are also crucial. Progress towards models that can fully understand scenes involving multiple people is hindered by a lack of sufficient annotated data for such high-level tasks. To address this challenge, we introduce Social-MAE, a simple yet effective transformer-based masked autoencoder framework for multi-person human motion data. The framework uses masked modeling to pre-train the encoder to reconstruct masked human joint trajectories, enabling it to learn generalizable and data efficient representations of motion in human crowded scenes. Social-MAE comprises a transformer as the MAE encoder and a lighter-weight transformer as the MAE decoder which operates on multi-person joints' trajectory in the frequency domain. After the reconstruction task, the MAE decoder is replaced with a task-specific decoder and the model is fine-tuned end-to-end for a variety of high-level social tasks. Our proposed model combined with our pre-training approach achieves the state-of-the-art results on various high-level social tasks, including multi-person pose forecasting, social grouping, and social action understanding. These improvements are demonstrated across four popular multi-person datasets encompassing both human 2D and 3D body pose.
BibTeX
@inproceedings{ehsanpour2024social,
  author = {Mahsa Ehsanpour and Ian Reid and Hamid Rezatofighi},
  title = {{Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
}
Figure