Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving

Zhiyu Huang, Haochen Liu, Jingda Wu, Chen Lv

Abstract

Making safe and human-like decisions is an essential capability of autonomous driving systems, and learning-based behavior planning presents a promising pathway toward achieving this objective. Distinguished from existing learning-based methods that directly output decisions, this work introduces a predictive behavior planning framework that learns to predict and evaluate from human driving data. This framework consists of three components: a behavior generation module that produces a diverse set of candidate behaviors in the form of trajectory proposals, a conditional motion prediction network that predicts future trajectories of other agents based on each proposal, and a scoring module that evaluates the candidate plans using maximum entropy inverse reinforcement learning (IRL). We validate the proposed framework on a large-scale real-world urban driving dataset through comprehensive experiments. The results show that the conditional prediction model can predict distinct and reasonable future trajectories given different trajectory proposals and the IRL-based scoring module can select plans that are close to human driving. The proposed framework outperforms other baseline methods in terms of similarity to human driving trajectories. Additionally, we find that the conditional prediction model improves both prediction and planning performance compared to the non-conditional model. Lastly, we note that learning the scoring module is crucial for aligning the evaluations with human drivers.

Introduction

In the quest for truly autonomous vehicles (AVs), the industry is moving beyond simple obstacle avoidance toward "socially compliant" driving. The goal is to develop cars that drive with the nuance and predictability of a human. Most current autonomous driving stacks treat prediction and planning as two separate, sequential steps: first, the car predicts where surrounding agents will go (usually assuming they are unaffected by the AV), and then it plans its own path.

This "passive" approach ignores the fundamental interactivity of driving; other drivers react to the AV's actions. For instance, if an AV initiates a lane change, a human driver behind it might decelerate to yield. Ignoring these interactions can lead to overly conservative or even dangerous behavior. Furthermore, manually tuning cost functions for speed, comfort, and safety is complex and often fails to capture subtle human preferences. This paper introduces a framework that combines Conditional Motion Prediction (CMP) with Inverse Reinforcement Learning (IRL) to address these challenges.

Methods

The proposed predictive behavior planning framework consists of three integrated components designed to "close the loop" between prediction and planning.

Overview of the proposed conditional predictive behavior planning framework.

1. Behavior Generation

The system generates a diverse set of 10–30 trajectory proposals (e.g., maintaining lane, changing lanes, or adjusting speed). These are constructed using polynomial curves in a Frenet frame—a coordinate system relative to the road center-line—to ensure the resulting paths are smooth and kinematically feasible.

2. Conditional Motion Prediction (CMP)

Using a Transformer-based neural network, the model predicts the future paths of all surrounding agents conditioned on a specific AV proposal. A key innovation is "Early Fusion," where the AV’s intended plan is injected into the network during the interaction-modeling stage. This allows the attention mechanisms to better capture how agents react to the AV's specific maneuvers compared to "Late Fusion" methods.

3. IRL Scoring

To select the optimal proposal, the system employs a cost function learned via Maximum Entropy Inverse Reinforcement Learning. By analyzing thousands of hours of human driving data from the Waymo Open Motion Dataset (WOMD), the model infers the weights for features such as travel efficiency, comfort (acceleration/jerk), and safety (headway and collision probability). This ensures the AV's choices align with human driving logic.

Results

The framework was validated using the WOMD, which contains over 570 hours of real-world urban driving data.

Qualitative results of the framework across various driving scenarios.

Prediction and Planning Performance

The study demonstrates that conditional prediction improves accuracy by approximately 10% over passive models. The framework achieved a minimum Final Displacement Error (minFDE) of 2.78 meters and an intention accuracy exceeding 95% for speed and 90% for lane choice. Notably, the conditional model performed nearly as well as an "Oracle" (which has perfect knowledge of the actual future), suggesting that prediction quality is a primary bottleneck in AV performance.

Human-Likeness and Efficiency

The IRL-based scoring module allowed the framework to outperform several baselines in terms of similarity to human trajectories. Unlike "black-box" end-to-end models, this framework remains interpretable; the learned weights of the cost function provide insight into the car's decision-making process. Despite the complexity of querying a Transformer for multiple proposals, the system is real-time capable, completing the pipeline in under 400ms on a standard consumer GPU.

Discussion and Conclusion

While the results are promising, the authors identify several areas for future work. The current validation is "open-loop," meaning it predicts on recorded data; "closed-loop" testing in simulators is needed to observe how interactions compound over time. Additionally, while the linear cost function aids interpretability, more complex structures might be required for rare edge cases. Finally, future research should focus on safety-critical scenarios, such as near-misses or erratic pedestrian behavior, which are less frequent in standard datasets.

In conclusion, by integrating conditional prediction with IRL, this framework provides a robust pathway toward autonomous vehicles that can navigate complex urban environments with human-like social intelligence.

Summary updated

Abstract

Making safe and human-like decisions is an essential capability of autonomous driving systems, and learning-based behavior planning presents a promising pathway toward achieving this objective. Distinguished from existing learning-based methods that directly output decisions, this work introduces a predictive behavior planning framework that learns to predict and evaluate from human driving data. This framework consists of three components: a behavior generation module that produces a diverse set of candidate behaviors in the form of trajectory proposals, a conditional motion prediction network that predicts future trajectories of other agents based on each proposal, and a scoring module that evaluates the candidate plans using maximum entropy inverse reinforcement learning (IRL). We validate the proposed framework on a large-scale real-world urban driving dataset through comprehensive experiments. The results show that the conditional prediction model can predict distinct and reasonable future trajectories given different trajectory proposals and the IRL-based scoring module can select plans that are close to human driving. The proposed framework outperforms other baseline methods in terms of similarity to human driving trajectories. Additionally, we find that the conditional prediction model improves both prediction and planning performance compared to the non-conditional model. Lastly, we note that learning the scoring module is crucial for aligning the evaluations with human drivers.
BibTeX
@article{huang2023conditional,
  author = {Zhiyu Huang and Haochen Liu and Jingda Wu and Chen Lv},
  title = {{Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=9zuYOhWWOGH},
}
Figure