Real Time Trajectory Prediction Using Deep Conditional Generative Models

Sebastián Gómez-González, Sergey Prokudin, Bernhard Schölkopf, Jan Peters

Introduction

In high-speed robotic applications, such as table tennis, the margin for error is measured in millimeters and milliseconds. For a robot to successfully intercept a fast-moving projectile, it must predict the object's future path with high precision and minimal latency. Traditional methods for trajectory prediction often struggle to balance physical accuracy with computational speed, particularly when dealing with complex, unobservable variables like aerodynamic spin. This paper presents a novel approach using a Trajectory Variational Auto-Encoder (TVAE) to provide robust, real-time predictions that outperform both classical physics-based models and standard recurrent neural networks (RNNs).

The Pitfalls of Traditional Modeling

Roboticists have historically relied on two primary paradigms for trajectory prediction, each with inherent limitations:

  1. Physics-Based Models: These models utilize differential equations to describe motion. While computationally efficient, they are often "blind" to hidden factors. In table tennis, a vision system can track a ball's position but cannot easily detect its spin. Because spin significantly alters the ball's flight path via the Magnus effect, physics-based models often diverge from reality as the flight progresses.
  2. Recursive Data-Driven Models (e.g., LSTMs): Long Short-Term Memory networks learn from data and can implicitly account for spin. However, they typically predict trajectories step-by-step. In this recursive process, a small error in the first predicted step is fed back into the model to predict the second step, causing errors to compound or "snowball" over time. This leads to significant inaccuracies in long-term forecasting.

The Trajectory Variational Auto-Encoder (TVAE)

To address these issues, the authors propose the Trajectory Variational Auto-Encoder (TVAE), a deep conditional generative model designed to predict a fixed window of the future trajectory in a single "one-shot" inference step.

Architecture

  • The Encoder: The encoder processes a sequence of past observations and maps them into a Gaussian-distributed latent space ($z$). To maintain high speed, the authors avoid recurrent layers. Instead, they use a feed-forward network with a masking strategy: variable-length inputs are zero-padded, and a binary mask is appended to indicate which data points are real observations. This allows the model to handle varying amounts of historical data without the sequential bottleneck of an RNN.
  • The Decoder: The decoder takes the latent variable $z$ along with the past observations to reconstruct the entire future trajectory up to a fixed time horizon ($T$). By outputting the whole path at once, the model bypasses the compounding error problem entirely.

Robustness and Uncertainty

The TVAE is trained using stochastic gradient variational Bayes. To ensure the model can handle the messiness of real-world robotics, the authors employed several data augmentation techniques during training, including randomly truncating trajectories and injecting noise or missing values. Furthermore, because the model is generative, it can sample multiple potential trajectories. The variance among these samples serves as a built-in uncertainty metric, allowing the robot to defer its movement if the prediction confidence is too low.

Experimental Evaluation

The model was evaluated using two distinct datasets:

  • Simulation Dataset: 2,000 trajectories generated via physics equations with added Gaussian noise to simulate sensor error.
  • Real-World Dataset: 649 trajectories captured via a 180-fps four-camera stereo vision system. This data included balls thrown by hand, launched by machines, and struck by rackets to ensure a wide variety of spin and velocity profiles.

Results and Performance

The TVAE demonstrated significant advantages over baseline models in three key areas:

  • Accuracy and Convergence: On real-world data, the TVAE's prediction error converged after observing only 40–50 frames. In contrast, an LSTM required approximately 150 frames to reach a comparable level of accuracy. This faster convergence grants the robot significantly more time to plan and execute a strike.
  • Computational Efficiency: The model achieved an inference latency of 8 to 10 ms. This speed is critical for real-time control loops, where the system must react to new visual data almost instantaneously.
  • Task Success: When integrated with a robot arm using Probabilistic Movement Primitives (ProMPs), the TVAE-based system achieved a 98.9% hitting rate, a notable improvement over the previous state-of-the-art baseline of 96.7%.

Conclusion

By moving away from recursive step-by-step prediction and embracing a generative, one-shot latent space approach, the TVAE provides a powerful solution for high-speed robotic tasks. While the model currently operates on a fixed prediction horizon, its ability to handle unobserved physical properties and provide real-time uncertainty estimates marks a significant step forward in dynamic robot interaction. Future work may explore hybrid architectures that combine the stability of TVAEs with the infinite horizons of state-space models.

Summary updated

Abstract

Data driven methods for time series forecasting that quantify uncertainty open new important possibilities for robot tasks with hard real time constraints, allowing the robot system to make decisions that trade off between reaction time and accuracy in the predictions. Despite the recent advances in deep learning, it is still challenging to make long term accurate predictions with the low latency required by real time robotic systems. In this paper, we propose a deep conditional generative model for trajectory prediction that is learned from a data set of collected trajectories. Our method uses encoder and decoder deep networks that maps complete or partial trajectories to a Gaussian distributed latent space and back, allowing for fast inference of the future values of a trajectory given previous observations. The encoder and decoder networks are trained using stochastic gradient variational Bayes. In the experiments, we show that our model provides more accurate long term predictions with a lower latency that popular models for trajectory forecasting like recurrent neural networks or physical models based on differential equations. Finally, we test our proposed approach in a robot table tennis scenario to evaluate the performance of the proposed method in a robotic task with hard real time constraints.
BibTeX
@article{gomezgonzalez2019real,
  author = {Sebastián Gómez-González and Sergey Prokudin and Bernhard Schölkopf and Jan Peters},
  title = {{Real Time Trajectory Prediction Using Deep Conditional Generative Models}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2019},
  doi = {10.1109/lra.2020.2966390},
}
Figure