TL;DR
BAL (Bidirectional Autoregression in Latent space) is a new self-supervised framework that learns human movement representations by predicting "what happens next" and "what happened before" in a semantically abstract latent space. By leveraging the natural temporal ordering of human behavior rather than artificial data augmentations, BAL achieves state-of-the-art performance in motion captioning, forecasting, and interpolation.
What Problem Does This Paper Address?
Learning to represent human movement from 3D pose sequences is a cornerstone of behavior modeling. Ideally, we want models to learn these representations through self-supervised learning (SSL) using the vast amounts of unlabeled pose data available.
However, current SSL methods rely on "artificial" supervision. Contrastive methods require hand-crafted data augmentations (like rotating or scaling a skeleton), while masked modeling requires choosing the "right" temporal scale for masking. These choices are often heuristic and require tedious hyperparameter tuning to work for specific tasks. Furthermore, predicting raw joint positions (pose-space prediction) often fails to capture high-level semantics because the model gets bogged down in low-level kinematic noise rather than understanding the underlying action (e.g., "sitting down").
Methods
The authors propose BAL, which shifts the focus from low-level joint prediction to high-level behavioral constraints.
- Latent Autoregression: Instead of predicting the next frame's joints, BAL divides a pose sequence into 4-second temporal chunks. An encoder compresses each chunk into a single latent token. The model then learns by predicting these tokens autoregressively. This mimics how humans anticipate behavior—we think in terms of "opening a fridge" rather than specific XYZ coordinates of a wrist.
- Bidirectional Training: BAL predicts in both temporal directions—forward and backward. This fully exploits the ordering constraints of human movement (e.g., you must reach for a handle before you pull it).
- Global-Local Decomposition: A major risk in latent-space prediction is "representation collapse," where the model learns to output a constant, meaningless value to make the prediction task trivially easy. BAL solves this by decomposing each target into a global component (the sequence-level mean) and a local component (the chunk-level residual).
- Mask-Free Inference: While trained with causal masks to ensure it learns predictive structure, at inference time, the masks are removed. This allows the model to extract representations using both past and future context in a single forward pass.
Datasets
The models were pretrained on a massive mixture of three daily activity datasets:
- AMD: 14.2 hours of household activities.
- Nymeria: ~300 hours of indoor/outdoor activities with text annotations.
- Embody 3D: ~500 hours of diverse behaviors, including multi-person interactions.
Key Insights and Ideas
- Intrinsic Supervision: The "aha moment" here is that the natural flow of time in human actions is a powerful, non-heuristic supervisory signal. We don't need to invent augmentations if we simply ask the model to understand the sequence of events.
- Mathematical Guardrails against Collapse: The authors provide a geometric proof that their global-local decomposition forces the local targets to retain "angular diversity." Even if the extractor starts to collapse, the local targets remain separated by at least 90°, preserving a gradient signal that keeps the model learning.
- Decoupling Extractor and Predictor: By using a shared representation extractor but separate direction-specific predictors, BAL ensures that the learned latent space is consistent regardless of whether the model is looking forward or backward.
Key Results
BAL was evaluated against several strong baselines (like A4Mer, BehaveMAE, and MotionBERT) across three downstream tasks:
- Motion Captioning: BAL produced significantly more accurate and semantically rich descriptions. For example, in Figure 2, BAL correctly identifies "assembling furniture" where baselines might only see "kneeling."
- Forecasting and Interpolation: BAL achieved the lowest errors (measured in MPJPE mm) on almost all metrics. In long-term forecasting (4 seconds ahead), it outperformed the next best baseline on the AMD dataset by a wide margin (90.5mm vs 136.1mm).
Limitations and Future Work
While BAL reduces reliance on many heuristics, the chunk length (set to 4 seconds) remains a hyperparameter. The authors note that while performance is stable across different lengths, shorter chunks favor pose-accurate tasks (forecasting), while longer chunks favor semantic tasks (captioning).
Future work could explore adaptive chunking to automatically discover the natural boundaries of actions. Additionally, while the paper focuses on 3D pose, the authors suggest these principles of bidirectional latent autoregression could be applied to other sequential modalities like video or audio. Finally, the authors acknowledge that improved movement representations could potentially be misused for unauthorized surveillance, highlighting the need for ethical deployment.
Summary updated