TL;DR
DIPP (Differentiable Integrated Prediction-Planning) is a novel framework that treats the motion planner as a differentiable optimizer, allowing for the joint training of prediction and planning modules. By enabling gradients to flow through the entire pipeline, the system learns optimal cost function weights from human driving data and produces "planning-centric" predictions that improve safety and human-like behavior in complex urban environments.
What Problem Does This Paper Address?
The traditional autonomous driving (AD) stack is often a "broken telephone" system. It typically consists of a forecasting module followed by a planning module. This separation leads to two major headaches:
- Prediction-Planning Mismatch: The prediction module is trained to minimize displacement error (ADE/FDE) without knowing how those predictions affect the AV’s actual plan. A small prediction error for a distant car might be irrelevant, while a small error for a merging car could be catastrophic.
- The "Weight Tuning" Nightmare: Planning relies on complex cost functions that balance safety, speed, and comfort. Manually tuning these weights is a laborious process of trial and error that rarely generalizes across all scenarios.
While end-to-end deep learning models try to solve this, they often lack the interpretability and safety guarantees of classical optimization-based planners.
Methods
The authors propose DIPP, which bridges the gap between classical optimization and deep learning.
- Neural Network Predictor: A Transformer-based architecture encodes agent history (via LSTM) and scene context (vectorized lanes and crosswalks). It outputs multi-modal joint trajectories for all agents in the scene.
- Differentiable Motion Planner: This is the core innovation. Instead of a black-box neural network, the planner is a differentiable nonlinear optimizer (using the Gauss-Newton algorithm). It explicitly optimizes the AV’s trajectory based on a cost function.
- Learnable Cost Function: The cost function includes terms for travel efficiency, ride comfort (acceleration and jerk), traffic rule adherence (lane centering, stop lines), and safety (distance to others). Crucially, the weights for these terms are learned parameters, optimized during training to minimize the difference between the AV’s plan and actual human driving.
- End-to-End Training: Because the optimizer is differentiable (implemented using the Theseus library), the "planning loss" can be back-propagated to update both the cost function weights and the weights of the prediction network.
Datasets
The framework was trained and validated on the Waymo Open Motion Dataset (WOMD), a large-scale, high-fidelity dataset of real-world urban driving. The authors used a subset of over 10,000 unique driving scenes, segmented into 7-second frames (2s history, 5s prediction/planning horizon).
Key Insights and Ideas
- Planning-Centric Prediction: By training jointly, the prediction module learns to prioritize accuracy in ways that benefit the downstream planner. The authors found that even if a model has slightly higher raw prediction error, it can result in a better plan if the errors are "planning-friendly."
- The Best of Both Worlds: DIPP maintains the structure of a classical planner (which provides safety and interpretability) but gains the data-driven adaptability of deep learning.
- Learnable Initialization: The paper highlights that providing a good "initial guess" to the optimizer is critical. DIPP uses the neural network to provide this initial plan, which significantly improves optimization stability and performance.
Key Results
The authors compared DIPP against several baselines, including Vanilla Imitation Learning (IL), Separated Prediction-Planning, and Reinforcement Learning (CQL).
- Safety & Rules: In open-loop tests, DIPP achieved the lowest collision rate (1.8%) and the lowest off-route rate (0.5%) compared to imitation learning baselines.
- Closed-Loop Robustness: In simulation, DIPP significantly outperformed the "Separated" pipeline. It handled complex interactions (like unprotected left turns and yielding to pedestrians) with a much lower failure rate (5% vs. 7% for separated and 40%+ for pure IL).
- Human-Likeness: The learned cost function weights resulted in trajectories that closely mimicked human acceleration and jerk profiles, whereas pure IL models often produced jerky, unrealistic movements.
Limitations and Future Work
- Computational Overhead: The primary bottleneck is the computation of the Jacobian during optimization. At test time, the system takes about 1.78 seconds per sample, which is far too slow for real-time deployment (typically requiring <0.1s). The authors suggest optimizing the computation pipeline as a next step.
- Uncertainty Handling: Currently, the planner only considers the "most likely" future predicted by the network. It does not yet perform full contingency planning that accounts for the multi-modality (multiple possible futures) of human behavior.
- Perception Assumptions: The model assumes perfect perception and localization. Integrating it with raw sensor data (LiDAR/Cameras) remains a challenge for future research.
Summary updated