Representing Spatial Trajectories as Distributions

Dídac Surís, Carl Vondrick

Researchers from Columbia University have developed a framework that represents partial observations of spatial trajectories—such as human motion—as probability distributions in a learned latent space. By treating trajectories as distributions rather than single points, the model naturally handles uncertainty, allows for continuous-time interpolation and extrapolation, and enables direct manipulation of motion attributes like speed and timing.

The Challenge of Modeling Motion

Predicting how an object moves, whether it is a gymnast mid-flip or a car approaching an intersection, is fraught with challenges. Trajectories are inherently continuous, yet we often observe them as discrete, irregular samples. Furthermore, observations are frequently partial due to occlusions or simply because the future hasn't happened yet.

A sequence of skeleton poses illustrating a complex, continuous spatial trajectory during a gymnastics movement.

Traditional autoregressive models predict the next step based on previous ones, but they often struggle with:

  1. Continuous time: They usually operate on fixed time steps.
  2. Global context: They lack a holistic representation of the entire motion.
  3. Uncertainty: A single "past" can lead to many plausible "futures" (multi-modality), which point-based representations struggle to capture.

To address this, the authors propose moving away from point-based latent spaces to a distribution-based latent space that characterizes the uncertainty of unobserved parts of a trajectory.

A Distributional Latent Space

The core of the framework is a "distributional" latent space. Instead of mapping an input segment to a single vector, an Encoder (Transformer) maps it to a probability distribution $Q$ in a latent space $\mathbb{R}^N$.

Key components include:

  • Latent Distributions: The authors experimented with Normal distributions and Box Embeddings. Box embeddings represent trajectories as high-dimensional hyper-rectangles; the "volume" of the box represents the set of all possible trajectories consistent with the observation.
  • Continuous Decoding: A Decoder (ResNet) takes a sample $z$ from the latent distribution and a continuous time value $t$ to produce a spatial coordinate. This allows the model to "query" the trajectory at any point in time, not just at predefined intervals.
  • Self-Supervised Learning: The model is trained using a triplet loss. It encourages distributions of segments from the same trajectory to overlap (high similarity) while pushing distributions from different trajectories apart.

Datasets and Keypoint Extraction

The model was evaluated on three human movement datasets where motion is complex and multi-modal: FineGym (high-quality gymnastics videos), Diving48 (competitive diving sequences), and FisV (figure skating videos).

2D skeleton keypoints extracted from a figure skating video using OpenPose, which serve as the spatial trajectory data for the model.

The researchers used OpenPose to extract 2D skeleton keypoints (25 joints) from these videos, treating the sequence of coordinates as the spatial trajectory to be modeled.

Key Insights and Capabilities

  • Intersection as Constraint: One of the most elegant ideas in the paper is that an intersection in the latent space corresponds to a "logical AND" in the observation space. If you have a distribution for the "past" and a distribution for the "future," their intersection in the latent space represents all trajectories that satisfy both observations.
  • Multi-modality by Design: Because the model represents a segment as a distribution, sampling different points from that distribution yields different plausible versions of the unobserved motion.
  • Temporal Editing: The latent space isn't just for reconstruction; it's interpretable. By moving along specific vectors in the latent space, the authors could modify the "pace" of a trajectory or shift its temporal offset without changing the fundamental motion.

Results and Future Work

The framework, dubbed TrajRep, was tested on future prediction, past prediction, and interpolation tasks.

Quantitatively, TrajRep outperformed strong baselines like VRNN and Trajectron++ in almost all categories. For example, on FineGym long sequences, TrajRep achieved a Mean Squared Error (MSE) of 8.50 for future prediction, compared to 9.72 for Trajectron++. Qualitatively, the model produces smooth, physically sensible interpolations. Unlike joint-level linear interpolation, TrajRep understands the "global" motion—for instance, realizing a gymnast must turn their body to connect two distant poses.

While powerful, the authors note a few areas for future improvement. The current model only looks at the trajectory itself and lacks environmental context (like the floor or a balance beam). Additionally, extending the experiments from 2D keypoints to 3D trajectories or raw video pixels is a natural next step. Overall, this work marks a significant step toward holistic motion understanding, treating movement as a unified, probabilistic entity.

Summary updated

Abstract

We introduce a representation learning framework for spatial trajectories. We represent partial observations of trajectories as probability distributions in a learned latent space, which characterize the uncertainty about unobserved parts of the trajectory. Our framework allows us to obtain samples from a trajectory for any continuous point in time—both interpolating and extrapolating. Our flexible approach supports directly modifying specific attributes of a trajectory, such as its pace, as well as combining different partial observations into single representations. Experiments show our method's superiority over baselines in prediction tasks.
BibTeX
@inproceedings{suris2022representing,
  author = {Dídac Surís and Carl Vondrick},
  title = {{Representing Spatial Trajectories as Distributions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=0WGlBa7pUV},
}
Figure