Abstract
Predicting the future states of surrounding traffic participants and planning a safe, smooth, and socially compliant trajectory is crucial for autonomous vehicles (AVs). Traditional systems often separate prediction and planning, leading to a "mismatch" where prediction errors are not weighted by their impact on the final plan. Furthermore, manually tuning cost functions for planning is notoriously difficult. We propose DIPP (Differentiable Integrated Prediction-Planning), a framework that uses a differentiable nonlinear optimizer as the motion planner. This allows the system to learn cost function weights from data and produces "planning-centric" predictions. Validated on the Waymo Open Motion Dataset, DIPP outperforms baselines in both open-loop and closed-loop tests, demonstrating improved safety and human-like driving behavior.
The Evolution of Prediction and Planning
The traditional autonomous driving stack typically follows a sequential pipeline: a forecasting module predicts agent trajectories, which are then passed to a motion planner. This "broken telephone" approach suffers from two primary issues:
- Prediction-Planning Mismatch: Forecasting modules are usually trained to minimize average displacement error. However, a small error regarding a nearby merging vehicle is far more critical than a large error for a distant parked car.
- Manual Weight Tuning: Planners rely on complex cost functions to balance safety, comfort, and efficiency. Manually tuning these weights is a laborious process that often fails to generalize.
While end-to-end neural models attempt to solve these issues, they often lack the interpretability and safety guarantees of classical optimization-based planners. DIPP bridges this gap by integrating a differentiable optimizer directly into the learning pipeline.
The DIPP Architecture
DIPP consists of two main components: a neural network predictor and a differentiable nonlinear motion planner.
Neural Network Predictor
The predictor uses a Transformer-based architecture to encode agent history and scene context (such as lane geometry). It outputs multi-modal joint trajectories for all surrounding agents. By training this module jointly with the planner, the network learns to prioritize "planning-centric" accuracy—focusing on predictions that most significantly impact the AV's safety and efficiency.
Differentiable Motion Planner
The core innovation is the use of a differentiable nonlinear optimizer (implemented using the Gauss-Newton algorithm). This planner explicitly optimizes the AV’s trajectory based on a learnable cost function. The cost function includes terms for efficiency, ride comfort (minimizing acceleration and jerk), traffic rule adherence (lane centering, stop lines), and safety (distance to others).
Because the optimizer is differentiable, gradients from the final planning loss can flow back through the entire system, updating both the cost function weights and the predictor's neural weights.
Experimental Results
DIPP was evaluated using the Waymo Open Motion Dataset (WOMD), involving over 10,000 unique real-world urban driving scenes.
Open-Loop Performance
In open-loop testing, DIPP achieved a collision rate of just 1.8% and an off-route rate of 0.5%, significantly lower than vanilla imitation learning baselines. The joint training approach ensured that the planner received predictions optimized for safe navigation rather than just raw displacement accuracy.
Closed-Loop Robustness
In closed-loop simulations, DIPP demonstrated superior handling of complex interactions, such as yielding to pedestrians and managing unprotected left turns. It maintained a much lower failure rate (5%) compared to a "separated" pipeline (7%) and pure imitation learning models (>40%). Furthermore, the learned cost weights produced trajectories that closely mimicked the smooth acceleration profiles of human drivers.
Discussion and Future Work
DIPP successfully combines the data-driven power of deep learning with the structural benefits of classical optimization. However, challenges remain. The primary bottleneck is computational overhead; current inference takes ~1.78s, which must be reduced for real-time deployment. Additionally, future iterations could incorporate full contingency planning to account for multiple possible futures predicted by the network. Despite these hurdles, DIPP represents a significant step toward more robust, human-like, and interpretable autonomous driving systems.
Summary updated