Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks

Nemanja Djuric, Vladan Radosavljevic, Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung-Han Lin, Jeff Schneider

Abstract

Predicting the future trajectories of traffic actors is a critical component of autonomous vehicle safety. This paper presents a deep learning framework developed by Uber ATG that treats motion prediction as a computer vision task. By rasterizing the actor's environment into top-down images and utilizing Convolutional Neural Networks (CNNs), the system captures complex road geometry and motion uncertainty. Experimental results on 240 hours of real-world driving data demonstrate that this approach significantly outperforms traditional physics-based models, reducing displacement error by over 60% while maintaining real-time performance.

Introduction

In the development of autonomous vehicles (AVs), perception—identifying what is currently in the environment—is only half the challenge. To navigate safely, an AV must excel at "prediction," anticipating where every nearby car, cyclist, and pedestrian will be in the coming seconds.

Traditional methods often rely on Kalman Filters, which use kinematic properties like velocity and acceleration to project a linear path forward. While accurate for very short durations, these models fail to account for environmental context. A vehicle does not simply move in a straight line; it follows lanes, obeys traffic signals, and reacts to other actors. Uber ATG’s research addresses this by treating motion prediction as a spatial reasoning problem, leveraging the power of Deep Convolutional Networks to interpret complex urban scenes.

Methodology: Rasterization and Deep Learning

The core innovation of this framework is the conversion of raw tracking and map data into a "rasterized" format suitable for computer vision models.

Rasterization Process

Instead of feeding raw coordinates into a model, the researchers create a top-down, bird’s-eye-view RGB image (a "raster") centered on the "actor of interest." This image encodes several layers of critical information:

  • Map Context: Road boundaries, crosswalks, and lanes are drawn as polygons, with lane directions encoded using colors from the HSV color space.
  • Actor History: To represent movement over time, the model draws the actor’s previous positions as a "trail" with a fading brightness effect. This allows a standard CNN to perceive velocity and intent without the need for complex recurrent architectures like LSTMs.
Top-down rasterized representation of the actor's environment, including map context and motion history.

Model Architecture

These rasterized images are fed into a standard CNN backbone (such as ResNet or MobileNet-v2). To enhance the model's predictive power, the high-level features extracted by the CNN are concatenated with a small vector containing the actor's current state—specifically velocity, acceleration, and heading change—before passing through final fully-connected layers.

The proposed model architecture combining rasterized features from a CNN with the actor's current state.

To account for the inherent unpredictability of driving, the model is trained to output a probability distribution rather than a single $(x, y)$ coordinate. By using a Negative Log-Likelihood (NLL) loss function, the network learns to estimate its own uncertainty (standard deviation) for every point along the predicted path.

Experimental Setup and Results

The authors utilized a massive proprietary dataset collected from Uber’s self-driving fleet, totaling 240 hours of real-world driving data from Pittsburgh, PA, and Phoenix, AZ. This dataset includes 7.8 million unique data points across diverse traffic conditions and complex urban maneuvers.

The deep learning approach significantly outperformed the traditional Unscented Kalman Filter (UKF) baseline across all metrics. The best-performing model (MobileNet-v2 with uncertainty) achieved an average displacement error of 0.71 meters over a 3-second horizon, compared to 2.36 meters for the UKF.

Key Insights

  • Uncertainty as a Regularizer: Interestingly, training the model to estimate its own uncertainty improved the accuracy of the point estimates themselves, suggesting that the NLL loss helps the model focus on the most probable outcomes.
  • Real-Time Constraints: Feed-forward CNNs were chosen over recurrent models to meet the strict 10ms latency window required for safe vehicle operation. The system was successfully deployed to Uber's fleet, performing batch inference in under 10ms.
  • Feature Importance: Ablation studies proved that both map context and fading history were critical. Without the map, the model could not "see" lanes, leading to significantly higher errors in complex turns.

Conclusion

By framing motion prediction as a vision problem, Uber ATG demonstrated a scalable way to incorporate environmental context into AV decision-making. While the current model predicts a fixed 3-second window and treats actors independently, future work aims to extend the prediction horizon and model multi-actor interactions to better anticipate how drivers influence one another in dense traffic.

Summary updated

BibTeX
@inproceedings{djuric2020motion,
  author = {Nemanja Djuric and Vladan Radosavljevic and Henggang Cui and Thi Nguyen and Fang-Chieh Chou and Tsung-Han Lin and Jeff Schneider},
  title = {{Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093332},
}
Figure