Conditional Predictive Behavior Planning With Inverse Reinforcement Learning for Human-Like Autonomous Driving

Zhiyu Huang, Haochen Liu, Jingda Wu, Chen Lv

In the quest for truly autonomous vehicles (AVs), the industry is moving beyond simple obstacle avoidance toward "socially compliant" driving. We want cars that don't just follow rules, but drive with the nuance and predictability of a human. A new paper by Zhiyu Huang and colleagues at Nanyang Technological University introduces a framework that achieves this by combining Conditional Motion Prediction (CMP) with Inverse Reinforcement Learning (IRL).

TL;DR

The researchers developed a behavior planning framework that predicts how surrounding traffic will react to specific AV actions and then uses a cost function learned from human driving data to select the best move. By "closing the loop" between prediction and planning, the system achieves significantly more human-like behavior on complex urban roads compared to traditional non-conditional models.

What Problem Does This Paper Address?

Most current autonomous driving stacks treat prediction and planning as two separate, sequential steps. First, the car predicts where everyone else will go (usually assuming they'll keep doing what they're doing), and then it plans its own path.

This "passive" approach has a major flaw: it ignores the fact that other drivers react to the AV. If an AV starts to change lanes, a human driver behind it might slow down to let it in. If the AV ignores this interaction, it might become overly conservative (waiting forever for a "perfect" gap) or even dangerous. Furthermore, manually tuning the "cost functions" that tell a car how to value speed vs. comfort vs. safety is incredibly difficult and often fails to capture the subtle preferences of human drivers.

Methods

The authors propose a three-part "Predictive Behavior Planning" framework:

  1. Behavior Generation: The system generates a diverse set of 10–30 "trajectory proposals" (e.g., stay in lane at 30mph, change left, slow down). These are created using polynomial curves in a Frenet frame (a coordinate system relative to the road center-line), ensuring they are smooth and feasible.
  2. Conditional Motion Prediction (CMP): This is the "brain" of the system. Using a Transformer-based neural network, the model predicts the future paths of all surrounding agents conditioned on a specific AV proposal.
  3. Early Fusion: A key technical contribution is "Early Fusion," where the AV’s intended plan is injected into the network at the interaction-modeling stage, rather than just at the end. This allows the model to better capture how agents react to the AV's specific maneuvers.
  4. IRL Scoring: To pick the best proposal, the system uses a cost function learned via Maximum Entropy Inverse Reinforcement Learning. Instead of engineers guessing weights for safety and comfort, the model looks at thousands of hours of human driving and "infers" the weights that make the AV's choices match human choices. The cost function considers travel efficiency, acceleration/jerk (comfort), and safety (headway and collision probability).

Datasets

The framework was trained and validated using the Waymo Open Motion Dataset (WOMD). This is one of the largest and most diverse real-world urban driving datasets available, containing over 100,000 unique scenes (570 hours of driving) across various cities. The researchers used over 113,000 data points for prediction training and a filtered subset of ~10,000 points for the IRL planning module.

Key Insights and Ideas

  • Interactivity is Measurable: The study proves that "conditional" prediction—knowing how others react to you—improves prediction accuracy by roughly 10% compared to passive models.
  • Early Fusion Matters: Fusing the AV's plan early in the neural network architecture is significantly more effective than "Late Fusion" (adding it at the end). It allows the attention mechanisms to better weigh the relationships between the AV's plan and the surrounding map/agents.
  • Closing the Gap to "Oracle": The researchers compared their model to an "Oracle" (which knows the actual future). Their conditional model performed nearly as well as the Oracle in planning tasks, suggesting that the bottleneck in AV performance is often the quality of the prediction, not just the planning logic.
  • Interpretability: Unlike "black-box" end-to-end deep learning models, this framework remains interpretable. You can look at the learned weights of the cost function to understand why the car chose to yield or overtake.

Key Results

  • Human-Likeness: The proposed framework outperformed several baselines (including standard neural network classifiers and model-based planners) in terms of similarity to human trajectories.
  • Accuracy: The model achieved a minFDE (minimum Final Displacement Error) of 2.78 meters and an intention accuracy of over 95% for speed and 90% for lane choice.
  • Efficiency: Despite the complexity of querying a Transformer for multiple trajectory proposals, the system is real-time capable, running the entire pipeline in under 400ms on a standard consumer GPU (RTX 3080).

Limitations and Future Work

  • Open-Loop Validation: The study primarily uses "open-loop" testing (predicting on recorded data). The authors acknowledge that the next step is "closed-loop" testing in a simulator to see how these interactions compound over longer periods.
  • Linear Cost Function: The IRL module currently uses a linear combination of features. While this aids interpretability, it might be too simple to capture the most complex "edge-case" human behaviors.
  • Safety-Critical Scenarios: The current dataset (Waymo) is largely composed of "normal" driving. Future work needs to investigate how this framework handles rare, high-risk scenarios like near-misses or erratic pedestrian behavior.

Reference: Huang, Z., Liu, H., Wu, J., & Lv, C. (2023). Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving. arXiv:2212.08787v3.

Summary updated

Abstract

Making safe and human-like decisions is an essential capability of autonomous driving systems, and learning-based behavior planning presents a promising pathway toward achieving this objective. Distinguished from existing learning-based methods that directly output decisions, this work introduces a predictive behavior planning framework that learns to predict and evaluate from human driving data. This framework consists of three components: a behavior generation module that produces a diverse set of candidate behaviors in the form of trajectory proposals, a conditional motion prediction network that predicts future trajectories of other agents based on each proposal, and a scoring module that evaluates the candidate plans using maximum entropy inverse reinforcement learning (IRL). We validate the proposed framework on a large-scale real-world urban driving dataset through comprehensive experiments. The results show that the conditional prediction model can predict distinct and reasonable future trajectories given different trajectory proposals and the IRL-based scoring module can select plans that are close to human driving. The proposed framework outperforms other baseline methods in terms of similarity to human driving trajectories. Additionally, we find that the conditional prediction model improves both prediction and planning performance compared to the non-conditional model. Lastly, we note that the learning of the scoring module is crucial for aligning the evaluations with human drivers.
BibTeX
@article{huang2023conditional,
  author = {Zhiyu Huang and Haochen Liu and Jingda Wu and Chen Lv},
  title = {{Conditional Predictive Behavior Planning With Inverse Reinforcement Learning for Human-Like Autonomous Driving}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=9zuYOhWWOGH},
}

Loading related content...

Figure