Abstract
Predicting the future trajectories of traffic actors is a critical component of autonomous vehicle safety. This paper presents a deep learning framework developed by Uber ATG that treats motion prediction as a computer vision task. By rasterizing the actor's environment into top-down images and utilizing Convolutional Neural Networks (CNNs), the system captures complex road geometry and motion uncertainty. Experimental results on 240 hours of real-world driving data demonstrate that this approach significantly outperforms traditional physics-based models, reducing displacement error by over 60% while maintaining real-time performance.
Introduction
In the development of autonomous vehicles (AVs), perception—identifying what is currently in the environment—is only half the challenge. To navigate safely, an AV must excel at "prediction," anticipating where every nearby car, cyclist, and pedestrian will be in the coming seconds.
Traditional methods often rely on Kalman Filters, which use kinematic properties like velocity and acceleration to project a linear path forward. While accurate for very short durations, these models fail to account for environmental context. A vehicle does not simply move in a straight line; it follows lanes, obeys traffic signals, and reacts to other actors. Uber ATG’s research addresses this by treating motion prediction as a spatial reasoning problem, leveraging the power of Deep Convolutional Networks to interpret complex urban scenes.
Methodology: Rasterization and Deep Learning
The core innovation of this framework is the conversion of raw tracking and map data into a "rasterized" format suitable for computer vision models.
Rasterization Process
Instead of feeding raw coordinates into a model, the researchers create a top-down, bird’s-eye-view RGB image (a "raster") centered on the "actor of interest." This image encodes several layers of critical information:
- Map Context: Road boundaries, crosswalks, and lanes are drawn as polygons, with lane directions encoded using colors from the HSV color space.
- Actor History: To represent movement over time, the model draws the actor’s previous positions as a "trail" with a fading brightness effect. This allows a standard CNN to perceive velocity and intent without the need for complex recurrent architectures like LSTMs.
Model Architecture
These rasterized images are fed into a standard CNN backbone (such as ResNet or MobileNet-v2). To enhance the model's predictive power, the high-level features extracted by the CNN are concatenated with a small vector containing the actor's current state—specifically velocity, acceleration, and heading change—before passing through final fully-connected layers.
To account for the inherent unpredictability of driving, the model is trained to output a probability distribution rather than a single $(x, y)$ coordinate. By using a Negative Log-Likelihood (NLL) loss function, the network learns to estimate its own uncertainty (standard deviation) for every point along the predicted path.
Experimental Setup and Results
The authors utilized a massive proprietary dataset collected from Uber’s self-driving fleet, totaling 240 hours of real-world driving data from Pittsburgh, PA, and Phoenix, AZ. This dataset includes 7.8 million unique data points across diverse traffic conditions and complex urban maneuvers.
The deep learning approach significantly outperformed the traditional Unscented Kalman Filter (UKF) baseline across all metrics. The best-performing model (MobileNet-v2 with uncertainty) achieved an average displacement error of 0.71 meters over a 3-second horizon, compared to 2.36 meters for the UKF.
Key Insights
- Uncertainty as a Regularizer: Interestingly, training the model to estimate its own uncertainty improved the accuracy of the point estimates themselves, suggesting that the NLL loss helps the model focus on the most probable outcomes.
- Real-Time Constraints: Feed-forward CNNs were chosen over recurrent models to meet the strict 10ms latency window required for safe vehicle operation. The system was successfully deployed to Uber's fleet, performing batch inference in under 10ms.
- Feature Importance: Ablation studies proved that both map context and fading history were critical. Without the map, the model could not "see" lanes, leading to significantly higher errors in complex turns.
Conclusion
By framing motion prediction as a vision problem, Uber ATG demonstrated a scalable way to incorporate environmental context into AV decision-making. While the current model predicts a fixed 3-second window and treats actors independently, future work aims to extend the prediction horizon and model multi-actor interactions to better anticipate how drivers influence one another in dense traffic.
Summary updated