BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations

Genki Kinoshita, Ko NishinoarXiv 2026 · arXiv

TL;DR

BAL (Bidirectional Autoregression in Latent space) is a new self-supervised framework that learns human movement representations by predicting "what happens next" and "what happened before" in a semantically abstract latent space. By leveraging the natural temporal ordering of human behavior rather than artificial data augmentations, BAL achieves state-of-the-art performance in motion captioning, forecasting, and interpolation.

What Problem Does This Paper Address?

Learning to represent human movement from 3D pose sequences is a cornerstone of behavior modeling. Ideally, we want models to learn these representations through self-supervised learning (SSL) using the vast amounts of unlabeled pose data available.

However, current SSL methods rely on "artificial" supervision. Contrastive methods require hand-crafted data augmentations (like rotating or scaling a skeleton), while masked modeling requires choosing the "right" temporal scale for masking. These choices are often heuristic and require tedious hyperparameter tuning to work for specific tasks. Furthermore, predicting raw joint positions (pose-space prediction) often fails to capture high-level semantics because the model gets bogged down in low-level kinematic noise rather than understanding the underlying action (e.g., "sitting down").

Methods

The authors propose BAL, which shifts the focus from low-level joint prediction to high-level behavioral constraints.

  1. Latent Autoregression: Instead of predicting the next frame's joints, BAL divides a pose sequence into 4-second temporal chunks. An encoder compresses each chunk into a single latent token. The model then learns by predicting these tokens autoregressively. This mimics how humans anticipate behavior—we think in terms of "opening a fridge" rather than specific XYZ coordinates of a wrist.
  2. Bidirectional Training: BAL predicts in both temporal directions—forward and backward. This fully exploits the ordering constraints of human movement (e.g., you must reach for a handle before you pull it).
  3. Global-Local Decomposition: A major risk in latent-space prediction is "representation collapse," where the model learns to output a constant, meaningless value to make the prediction task trivially easy. BAL solves this by decomposing each target into a global component (the sequence-level mean) and a local component (the chunk-level residual).
  4. Mask-Free Inference: While trained with causal masks to ensure it learns predictive structure, at inference time, the masks are removed. This allows the model to extract representations using both past and future context in a single forward pass.
Figure 1: BAL learns movement representations from 3D pose sequences by bidirectional latent-space autoregressive prediction. The Encoder consolidates each chunk into a latent token, and a shared LatentFormer processes these tokens in both directions using causal and reversed causal masks. Two direction-specific predictors predict local components of adjacent latent tokens, while the global component is predicted from a learnable [CLS] token. At inference time, the LatentFormer extracts chunk representations without causal masking, enabling bidirectional contextual inference.

Datasets

The models were pretrained on a massive mixture of three daily activity datasets:

  • AMD: 14.2 hours of household activities.
  • Nymeria: ~300 hours of indoor/outdoor activities with text annotations.
  • Embody 3D: ~500 hours of diverse behaviors, including multi-person interactions.

Key Insights and Ideas

  • Intrinsic Supervision: The "aha moment" here is that the natural flow of time in human actions is a powerful, non-heuristic supervisory signal. We don't need to invent augmentations if we simply ask the model to understand the sequence of events.
  • Mathematical Guardrails against Collapse: The authors provide a geometric proof that their global-local decomposition forces the local targets to retain "angular diversity." Even if the extractor starts to collapse, the local targets remain separated by at least 90°, preserving a gradient signal that keeps the model learning.
  • Decoupling Extractor and Predictor: By using a shared representation extractor but separate direction-specific predictors, BAL ensures that the learned latent space is consistent regardless of whether the model is looking forward or backward.

Key Results

BAL was evaluated against several strong baselines (like A4Mer, BehaveMAE, and MotionBERT) across three downstream tasks:

  • Motion Captioning: BAL produced significantly more accurate and semantically rich descriptions. For example, in Figure 2, BAL correctly identifies "assembling furniture" where baselines might only see "kneeling."
  • Forecasting and Interpolation: BAL achieved the lowest errors (measured in MPJPE mm) on almost all metrics. In long-term forecasting (4 seconds ahead), it outperformed the next best baseline on the AMD dataset by a wide margin (90.5mm vs 136.1mm).
Figure 2: Motion captioning results. BAL generates more accurate descriptions than the baselines, suggesting that its representations capture the semantics of the movement more effectively.

Limitations and Future Work

While BAL reduces reliance on many heuristics, the chunk length (set to 4 seconds) remains a hyperparameter. The authors note that while performance is stable across different lengths, shorter chunks favor pose-accurate tasks (forecasting), while longer chunks favor semantic tasks (captioning).

Future work could explore adaptive chunking to automatically discover the natural boundaries of actions. Additionally, while the paper focuses on 3D pose, the authors suggest these principles of bidirectional latent autoregression could be applied to other sequential modalities like video or audio. Finally, the authors acknowledge that improved movement representations could potentially be misused for unauthorized surveillance, highlighting the need for ethical deployment.

Summary updated

Abstract

Learning semantically meaningful human movement representations from 3D pose sequences is essential for human behavior modeling. Existing self-supervised methods derive supervision from artificial assumptions, e.g., invariance to augmentation within a certain range or recoverability of masked regions at a certain temporal scale, introducing heuristic hyperparameters that make the learned semantics sensitive to their settings. We propose BAL, a self-supervised framework that instead exploits supervision intrinsic to human movement: the natural ordering structure of everyday behavior. BAL autoregressively predicts in a semantically abstract latent space, analogous to how humans anticipate others' actions rather than their precise joint configurations. Bidirectional autoregressive prediction, both forward and backward in time, fully leverages these ordering constraints while enabling representation extraction using both past and future context at inference. To suppress representation collapse, BAL decomposes each prediction target into a global component capturing sequence-level semantics and a local component capturing context-independent movement. We prove that the local targets retain nontrivial angular diversity unless the representations are completely collapsed, preserving a meaningful training signal. Experiments on large-scale datasets demonstrate that BAL's representations are highly effective for motion captioning, forecasting, and interpolation, outperforming baselines on most evaluation metrics.
BibTeX
@misc{kinoshita2026bal,
  author = {Genki Kinoshita and Ko Nishino},
  title = {{BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations}},
  year = {2026},
  eprint = {2611.18559},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18559},
}
Figure