Abstract
Predicting the future motion of traffic actors is critical for safe autonomous vehicle operation. This paper introduces a deep learning framework that represents the environment as a multi-channel bird's-eye view raster image, incorporating high-definition map data and actor history. By leveraging convolutional neural networks and LSTM decoders, the system provides accurate 3-second trajectory predictions along with calibrated uncertainty estimates. Evaluated on a large-scale real-world dataset, the model significantly outperforms physics-based baselines and demonstrates real-time feasibility on production autonomous platforms.
Introduction
To navigate safely, an autonomous vehicle must accurately predict the trajectories of surrounding actors—cars, pedestrians, and cyclists—over a short-term horizon. Traditional motion prediction often relies on physics-based models like Unscented Kalman Filters (UKF), which excel at very short-term projections but lack the environmental context necessary to follow complex lane geometries. Conversely, heuristic models that manually associate actors with lanes can be brittle when faced with non-standard maneuvers. This work proposes a unified deep learning approach that treats motion prediction as a computer vision task, allowing the model to learn spatial context and behavioral uncertainty directly from data.
Methodology
Rasterization as a Universal Interface
The core of the proposed framework is a multi-channel rasterization process. Instead of processing raw coordinates, the system generates a bird's-eye view (BEV) image that encodes the static environment and dynamic actor states. High-definition map features, such as road boundaries and crosswalks, are rendered into the image. Lane directions are encoded using the HSV color space, where hue represents the flow angle, providing the network with explicit directional cues.
To capture temporal dynamics within a static image, the history of the target actor is rendered as a series of bounding boxes with decaying brightness. This "fading" effect allows a standard 2D Convolutional Neural Network (CNN) to perceive velocity and acceleration without the need for more computationally expensive 3D convolutions.
Model Architecture and Uncertainty
The architecture utilizes a CNN backbone (e.g., ResNet or MobileNet-v2) to extract spatial features from the raster input. These features are concatenated with the actor's current state (velocity, acceleration) and passed through a decoder to generate the predicted trajectory. The authors found that an LSTM-based decoder provided the best performance for sequential coordinate generation.
Crucially, the model is designed to be uncertainty-aware. Rather than outputting a single deterministic path, the network predicts a standard deviation ($\sigma$) for each $(x, y)$ coordinate. The model is trained using a Negative Log-Likelihood (NLL) loss, assuming a half-normal distribution for displacement errors. This approach not only provides a confidence measure for the motion planner but also improves overall convergence by effectively down-weighting ambiguous training examples.
Experimental Results
Dataset
The model was trained on a massive real-world dataset comprising 240 hours of driving data from Pittsburgh and Phoenix. This resulted in 7.8 million unique examples, sampled at 10Hz to match the frequency of onboard tracking systems.
Performance Metrics
The deep learning framework demonstrated significant improvements over traditional baselines:
- Accuracy: The MobileNet-v2 + LSTM configuration achieved an Average Displacement Error (ADE) of 0.62 meters at a 3-second horizon, compared to 1.46 meters for the UKF baseline.
- Lane Adherence: Cross-track error was reduced by over 75%, indicating the model's superior ability to utilize map context.
- Efficiency: The system is optimized for production, performing batch inference for 32 objects in approximately 10ms on an onboard GPU.
- Calibration: Reliability diagrams confirmed that the predicted uncertainty closely matches observed error distributions, making the output highly actionable for downstream planning.
Conclusion
This research demonstrates that representing the driving environment as a rasterized image allows deep learning models to effectively integrate map context and temporal history for motion prediction. The resulting system is more accurate than physics-based models, provides well-calibrated uncertainty estimates, and meets the strict latency requirements of real-world autonomous driving. Future work will focus on multi-modal predictions to handle ambiguous scenarios like intersection forks and longer-term reasoning about actor-to-actor interactions.
Summary updated