TL;DR
Social-MAE is a self-supervised framework that learns rich representations of multi-person motion by reconstructing masked joint trajectories in the frequency domain. By pre-training on unlabeled motion data, it achieves state-of-the-art performance across diverse high-level social tasks—such as pose forecasting, social grouping, and action detection—while demonstrating superior data efficiency.
What Problem Does This Paper Address?
While computer vision has made massive strides in detecting and tracking individuals, truly "understanding" a scene requires moving into the social realm. We need models that can predict where people will move next, identify who is interacting with whom (social grouping), and recognize complex social actions like "listening to someone."
The primary bottleneck for these high-level tasks is a lack of annotated data. Labeling social interactions is significantly more labor-intensive and subjective than drawing bounding boxes. Consequently, models trained purely via supervised learning often struggle to generalize to new environments. Social-MAE addresses this by leveraging self-supervised learning, allowing models to learn the underlying "grammar" of human motion and interaction from vast amounts of unlabeled pose data before being fine-tuned for specific tasks.
Methods
The authors introduce an asymmetric Masked Autoencoder (MAE) architecture specifically tailored for spatio-temporal motion data.
- Input & Frequency Domain: Instead of feeding raw Cartesian coordinates $(x, y, z)$ directly into the model, the joint trajectories are transformed into the frequency domain using a Discrete Cosine Transform (DCT). This captures the periodic and continuous nature of human movement more effectively than raw sequences.
- Tube Masking: To prevent the model from "cheating" by interpolating between adjacent frames, the authors use a strategy called "Tube Masking." They randomly select a subset of joints and mask their entire trajectory across the time window.
- Asymmetric Architecture: The framework consists of a deep Transformer encoder and a shallow, lightweight Transformer decoder. The encoder only processes the visible (unmasked) joint trajectories, making it highly efficient. The decoder then attempts to reconstruct the full set of trajectories in the DCT space.
- Downstream Adaptation: Once pre-trained, the decoder is discarded. The pre-trained encoder is then coupled with task-specific decoders for three distinct social tasks:
- Social Pose Forecasting: Predicting future joint positions.
- Social Grouping: Determining if individuals belong to the same functional group.
- Social Action Understanding: Recognizing individual and interaction-based actions.
Datasets
The model was evaluated across four major datasets covering both 2D and 3D human poses:
- 3D Motion Forecasting: Evaluated on 3DPW, AMASS, CMU-Mocap, and MuPoTS-3D. The authors even synthesized multi-person scenes by mixing single-person sequences from AMASS.
- Social Interaction: Evaluated on JRDB-Act, a large-scale dataset captured by a mobile robot in crowded environments, containing over 2.8 million social action labels.
Key Insights and Ideas
- The Power of Sparsity: Similar to findings in the image and video domains, the authors found that masking a significant portion of the data (50%) forces the encoder to learn high-level structural and social dependencies rather than just local correlations.
- Motion is Redundant: The optimal masking ratio for motion (50%) is lower than for video (90%). This suggests that while motion data is redundant, it is less so than dense pixel grids, requiring a more careful balance to ensure the reconstruction task remains challenging but solvable.
- Social Context Matters: By processing all individuals in a scene simultaneously within the transformer, the encoder naturally learns to represent how one person's motion influences another's—a critical requirement for social understanding.
Key Results
Social-MAE sets a new state-of-the-art (SOTA) across all tested tasks:
- Pose Forecasting: On the SoMoF benchmark, Social-MAE achieved an overall VIM (Visibility-Ignored Metric) of 48.6, outperforming the previous SOTA (49.4).
- Social Grouping: On the JRDB-Act test set, it achieved a mean Average Precision (mAP) of 32.4, significantly higher than previous methods that relied on visual data or complex hand-crafted rules.
- Action Detection: It outperformed existing models on JRDB-Act, reaching an mAP of 11.2 on the test set, proving that motion trajectories alone can be more informative for social actions than raw video frames in some contexts.
- Data Efficiency: Ablation studies showed that the pre-trained model is much more resilient to reductions in labeled training data compared to models trained from scratch.
Limitations and Future Work
The authors identify a few key areas for improvement:
- Visual Ambiguity: The model occasionally produces false positives for actions like "talking to someone" when people are simply moving in close proximity. The authors suggest that incorporating visual features (pixels) alongside pose data could help resolve these social nuances.
- 2D vs. 3D Grouping: Currently, grouping failures often occur because the model relies on 2D trajectories, which can be deceptive in crowded scenes. Moving toward 3D data for social grouping is a proposed future direction.
- Identity Sensitivity: While using pose data minimizes the risk of exposing sensitive identity details compared to video, the authors acknowledge that behavioral patterns themselves can be sensitive.
Summary updated