Differentiable Integrated Motion Prediction and Planning With Learnable Cost Function for Autonomous Driving

Zhiyu Huang, Haochen Liu, Jingda Wu, Chen Lv

TL;DR

DIPP (Differentiable Integrated Prediction-Planning) is a novel framework that treats the motion planner as a differentiable optimizer, allowing for the joint training of prediction and planning modules. By enabling gradients to flow through the entire pipeline, the system learns optimal cost function weights from human driving data and produces "planning-centric" predictions that improve safety and human-like behavior in complex urban environments.

What Problem Does This Paper Address?

The traditional autonomous driving (AD) stack is often a "broken telephone" system. It typically consists of a forecasting module followed by a planning module. This separation leads to two major headaches:

  1. Prediction-Planning Mismatch: The prediction module is trained to minimize displacement error (ADE/FDE) without knowing how those predictions affect the AV’s actual plan. A small prediction error for a distant car might be irrelevant, while a small error for a merging car could be catastrophic.
  2. The "Weight Tuning" Nightmare: Planning relies on complex cost functions that balance safety, speed, and comfort. Manually tuning these weights is a laborious process of trial and error that rarely generalizes across all scenarios.

While end-to-end deep learning models try to solve this, they often lack the interpretability and safety guarantees of classical optimization-based planners.

Methods

The authors propose DIPP, which bridges the gap between classical optimization and deep learning.

  • Neural Network Predictor: A Transformer-based architecture encodes agent history (via LSTM) and scene context (vectorized lanes and crosswalks). It outputs multi-modal joint trajectories for all agents in the scene.
  • Differentiable Motion Planner: This is the core innovation. Instead of a black-box neural network, the planner is a differentiable nonlinear optimizer (using the Gauss-Newton algorithm). It explicitly optimizes the AV’s trajectory based on a cost function.
  • Learnable Cost Function: The cost function includes terms for travel efficiency, ride comfort (acceleration and jerk), traffic rule adherence (lane centering, stop lines), and safety (distance to others). Crucially, the weights for these terms are learned parameters, optimized during training to minimize the difference between the AV’s plan and actual human driving.
  • End-to-End Training: Because the optimizer is differentiable (implemented using the Theseus library), the "planning loss" can be back-propagated to update both the cost function weights and the weights of the prediction network.

Datasets

The framework was trained and validated on the Waymo Open Motion Dataset (WOMD), a large-scale, high-fidelity dataset of real-world urban driving. The authors used a subset of over 10,000 unique driving scenes, segmented into 7-second frames (2s history, 5s prediction/planning horizon).

Key Insights and Ideas

  • Planning-Centric Prediction: By training jointly, the prediction module learns to prioritize accuracy in ways that benefit the downstream planner. The authors found that even if a model has slightly higher raw prediction error, it can result in a better plan if the errors are "planning-friendly."
  • The Best of Both Worlds: DIPP maintains the structure of a classical planner (which provides safety and interpretability) but gains the data-driven adaptability of deep learning.
  • Learnable Initialization: The paper highlights that providing a good "initial guess" to the optimizer is critical. DIPP uses the neural network to provide this initial plan, which significantly improves optimization stability and performance.

Key Results

The authors compared DIPP against several baselines, including Vanilla Imitation Learning (IL), Separated Prediction-Planning, and Reinforcement Learning (CQL).

  • Safety & Rules: In open-loop tests, DIPP achieved the lowest collision rate (1.8%) and the lowest off-route rate (0.5%) compared to imitation learning baselines.
  • Closed-Loop Robustness: In simulation, DIPP significantly outperformed the "Separated" pipeline. It handled complex interactions (like unprotected left turns and yielding to pedestrians) with a much lower failure rate (5% vs. 7% for separated and 40%+ for pure IL).
  • Human-Likeness: The learned cost function weights resulted in trajectories that closely mimicked human acceleration and jerk profiles, whereas pure IL models often produced jerky, unrealistic movements.

Limitations and Future Work

  • Computational Overhead: The primary bottleneck is the computation of the Jacobian during optimization. At test time, the system takes about 1.78 seconds per sample, which is far too slow for real-time deployment (typically requiring <0.1s). The authors suggest optimizing the computation pipeline as a next step.
  • Uncertainty Handling: Currently, the planner only considers the "most likely" future predicted by the network. It does not yet perform full contingency planning that accounts for the multi-modality (multiple possible futures) of human behavior.
  • Perception Assumptions: The model assumes perfect perception and localization. Integrating it with raw sensor data (LiDAR/Cameras) remains a challenge for future research.

Summary updated

Abstract

Predicting the future states of surrounding traffic participants and planning a safe, smooth, and socially compliant trajectory accordingly are crucial for autonomous vehicles (AVs). There are two major issues with the current autonomous driving system: the prediction module is often separated from the planning module, and the cost function for planning is hard to specify and tune. To tackle these issues, we propose a differentiable integrated prediction and planning (DIPP) framework that can also learn the cost function from data. Specifically, our framework uses a differentiable nonlinear optimizer as the motion planner, which takes as input the predicted trajectories of surrounding agents given by the neural network and optimizes the trajectory for the AV, enabling all operations to be differentiable, including the cost function weights. The proposed framework is trained on a large-scale real-world driving dataset to imitate human driving trajectories in the entire driving scene and validated in both open-loop and closed-loop manners. The open-loop testing results reveal that the proposed method outperforms the baseline methods across a variety of metrics and delivers planning-centric prediction results, allowing the planning module to output trajectories close to those of human drivers. In closed-loop testing, the proposed method outperforms various baseline methods, showing the ability to handle complex urban driving scenarios and robustness against the distributional shift. Importantly, we find that joint training of planning and prediction modules achieves better performance than planning with a separate trained prediction module in both open-loop and closed-loop tests. Moreover, the ablation study indicates that the learnable components in the framework are essential to ensure planning stability and performance. Code and Supplementary Videos are available at https://mczhi.github.io/DIPP/.
BibTeX
@article{huang2024differentiable,
  author = {Zhiyu Huang and Haochen Liu and Jingda Wu and Chen Lv},
  title = {{Differentiable Integrated Motion Prediction and Planning With Learnable Cost Function for Autonomous Driving}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=z7Z8nCoPBT},
}

Loading related content...

Figure