Abstract
We introduce a representation learning framework for spatial trajectories. We represent partial observations of trajectories as probability distributions in a learned latent space, which characterize the uncertainty about unobserved parts of the trajectory. Our framework allows us to obtain samples from a trajectory for any continuous point in time, both interpolating and extrapolating. Our flexible approach supports directly modifying specific attributes of a trajectory, such as its pace, as well as combining different partial observations into single representations. Experiments show our method's advantage over baselines in prediction tasks.
Introduction
Spatial trajectories—the paths objects take through space and time—are fundamental to understanding the visual world, from a gymnast’s floor routine to the trajectory of a rolling ball. However, modeling these paths is notoriously difficult due to three primary challenges:
- Uncertainty: A short segment of motion could belong to many different full trajectories. A mid-air jump might lead to a successful landing or a fall, and a model must account for these varied possibilities.
- Continuity: While trajectories are inherently continuous, observations are typically discrete, irregularly sampled, and often plagued by occlusions.
- Global Representation: Most existing models are autoregressive, predicting the next step based on the previous one. This makes it difficult to reason about the trajectory as a holistic object or to perform non-sequential tasks like interpolation (filling in the middle).
To address these issues, we propose TrajRep, a framework that represents spatial trajectories as probability distributions in a learned latent space rather than simple sequences of points.
Methodology
The TrajRep framework maps partial observations to latent distributions, naturally capturing uncertainty and enabling continuous-time querying.
The Encoder ($\Theta$)
The encoder uses a Transformer-based architecture to process a sequence of observed points (a segment) and map it to a probability distribution $Q$ in a latent space. This mapping ensures that even limited data can be represented within a broader context of potential outcomes.
Latent Distributions
We explore two distinct types of latent representations:
- Normal Distributions: Represented by a mean and variance to capture the central tendency and spread of possible trajectories.
- Box Embeddings: These represent trajectories as $N$-dimensional hyperrectangles. This approach is particularly powerful because the intersection of two boxes in latent space represents the set of trajectories that satisfy the constraints of both observed segments simultaneously.
The Decoder ($\Phi$)
The decoder is a ResNet-based "neural implicit" function. It takes a sampled latent vector $z$ and a continuous time value $t$ as input to output a spatial coordinate. Because $t$ is continuous, the model can query the trajectory at any arbitrary temporal resolution, effectively interpolating or extrapolating beyond the training data.
Self-Supervised Training
The model is trained using a combination of a triplet loss and a reconstruction loss. The triplet loss encourages segments from the same trajectory to have overlapping latent distributions (positives) while pushing segments from different trajectories apart (negatives). The reconstruction loss ensures that decoded points accurately match the input observations.
Experimental Setup
We evaluated TrajRep on three human movement datasets, using OpenPose to extract 2D skeleton keypoints (25 joints):
- FineGym: High-quality gymnastics videos.
- Diving48: Competitive diving sequences.
- FisV: Figure skating videos.
Testing was conducted on both "short" sequences (up to 10 frames) and "long" sequences (up to 30 frames) to measure the model's ability to generalize across complex, high-variance human motions.
Key Insights
- Uncertainty as Volume: A partial observation defines a region of possibility. As more segments are observed and their latent distributions are intersected, the volume of uncertainty shrinks, refining the prediction.
- Intersection as Constraint Satisfaction: In latent space, the intersection of a "past" distribution and a "future" distribution yields a representation of the "middle." This allows the model to perform interpolation at inference time without explicit training for that task.
- Temporal Editing: The structured latent space allows for the identification of vectors corresponding to physical attributes. By shifting a latent vector along these axes, we can "edit" a trajectory—for instance, increasing the pace of a gymnast's routine or shifting the motion's temporal offset.
Results and Discussion
TrajRep consistently outperformed strong baselines, including VRNN and Trajectron++, across all datasets. On FineGym long sequences, for example, the model reduced Mean Squared Error (MSE) from approximately 9.7 to 8.5.
Beyond accuracy, the framework offers unique functional advantages. Unlike autoregressive models limited by input frame rates, TrajRep generates smooth motions at higher temporal resolutions than the training data. Furthermore, by sampling different points from the latent distribution, the model generates multiple plausible futures, capturing the inherent multi-modality of human motion.
Limitations and Future Work
While effective, the current framework focuses on 2D keypoints; scaling to high-dimensional "pixel-space" (raw video) remains a significant challenge. Additionally, for very long-term predictions, the model tends to revert to an "average" trajectory, losing some idiosyncratic detail. Future iterations could incorporate environmental context—such as the location of a balance beam—to further constrain latent distributions and improve long-term accuracy.
Summary updated