Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learning

Kasra Rezaee, Peyman Yadmellat, Simon Chamorro

Abstract

Motion planning under uncertainty is one of the main challenges in developing autonomous driving vehicles. In this work, we focus on the uncertainty in sensing and perception, resulted from a limited field of view, occlusions, and sensing range. This problem is often tackled by considering hypothetical hidden objects in occluded areas or beyond the sensing range to guarantee passive safety. However, this may result in conservative planning and expensive computation, particularly when numerous hypothetical objects need to be considered. We propose a reinforcement learning (RL) based solution to manage uncertainty by optimizing for the worst case outcome. This approach is in contrast to traditional RL, where the agents try to maximize the average expected reward. The proposed approach is built on top of the Distributional RL with its policy optimization maximizing the stochastic outcomes' lower bound. This modification can be applied to a range of RL algorithms. As a proof-of-concept, the approach is applied to two different RL algorithms, Soft Actor-Critic and DQN. The approach is evaluated against two challenging scenarios of pedestrians crossing with occlusion and curved roads with a limited field of view. The algorithm is trained and evaluated using the SUMO traffic simulator. The proposed approach yields much better motion planning behavior compared to conventional RL algorithms and behaves comparably to humans driving style.

Introduction

Motion planning for autonomous vehicles (AVs) is inherently difficult due to environmental uncertainty, particularly from occlusions like parked trucks or sharp curves. Traditional robotics approaches often rely on "ghost" objects to represent potential hazards in unobserved areas. While safe, this can lead to overly conservative "frozen robot" behavior and high computational costs. Conversely, standard reinforcement learning (RL) agents are typically "optimists" that maximize average expected rewards. This strategy is dangerous in safety-critical domains; if a blind corner is clear 99% of the time, an optimistic agent may speed through it, failing catastrophically during the rare 1% event when a pedestrian is present.

Methodology

This paper introduces Conservative Reinforcement Learning (CQR), a framework that leverages distributional RL to optimize for worst-case scenarios. Instead of learning a single scalar value for a state-action pair, the agent uses Quantile Regression to estimate the entire probability distribution of potential rewards. The core innovation is modifying the objective function to maximize the first quantile ($q_1$), effectively prioritizing the lower bound of potential outcomes.

The authors integrated this conservative approach into two architectures: CQR-DQN (value-based) and CQR-SAC (policy-based). They further distinguished between trajectory-based ($\tau$) and policy-based ($\pi$) evaluation, finding that the latter—which assumes the vehicle can adapt its strategy in future steps—offered superior flexibility.

Experiments were conducted in the SUMO (Simulation of Urban MObility) simulator across two high-stakes scenarios:

  1. Pedestrian Crossing: A large parked object obscures pedestrians at a crosswalk.
  2. Curved Road: A sharp curve with a barrier blocking the line of sight, occasionally hiding a stalled vehicle.

A curriculum learning approach was employed, gradually increasing occlusion proximity to the road to teach the agent prudence.

Results and Analysis

The conservative policy-based models significantly outperformed standard RL baselines in both safety and behavioral sophistication.

Emergent Defensive Driving

A key finding was the emergence of human-like defensive behaviors without explicit rule-based programming. Conservative agents learned to slow down significantly before blind spots, maintaining a "safety buffer" that allowed them to react to sudden hazards.

Comparison of standard SAC and conservative CQR-SAC behavior. The CQR-SAC agent (bottom) slows down (lighter colors) and positions itself further left to improve its field of view around the occlusion.

Furthermore, the agents exhibited "peeking" behavior: when approaching a right-hand occlusion, they would maneuver to the left side of the lane to increase their field of view. This sophisticated tactic is common in human driving but absent in standard RL agents, which tend to stay centered or ignore the risk entirely.

Quantitative Performance

Quantitatively, the CQR models achieved drastic reductions in collision rates. In the curved road scenario, standard SAC suffered a 37.56% collision rate, whereas CQR-SAC$_\pi$ reduced this to 3.71%.

Performance comparison across metrics. CQR-SAC$_\pi$ shows significantly lower collision rates and more cautious average speeds compared to standard SAC and DQN.

The data suggests that while conservative agents may have lower average rewards due to slower travel times, their survival rate is vastly superior. This highlights the "average reward trap" in standard RL, where high mean performance can mask fatal edge-case failures.

Conclusion

The CQR framework demonstrates that optimizing for the lower bound of reward distributions leads to safer, more human-like motion planning in occluded environments. Future work will focus on transitioning from simulation to real-world environments with sensor noise and incorporating dynamic agents, where the ego-vehicle must infer occluded states by observing the behavior of other drivers.

Summary updated

Abstract

Motion planning under uncertainty is one of the main challenges in developing autonomous driving vehicles. In this work, we focus on the uncertainty in sensing and perception, resulted from a limited field of view, occlusions, and sensing range. This problem is often tackled by considering hypothetical hidden objects in occluded areas or beyond the sensing range to guarantee passive safety. However, this may result in conservative planning and expensive computation, particularly when numerous hypothetical objects need to be considered. We propose a reinforcement learning (RL) based solution to manage uncertainty by optimizing for the worst case outcome. This approach is in contrast to traditional RL, where the agents try to maximize the average expected reward. The proposed approach is built on top of the Distributional RL with its policy optimization maximizing the stochastic outcomes' lower bound. This modification can be applied to a range of RL algorithms. As a proof-of-concept, the approach is applied to two different RL algorithms, Soft Actor-Critic and DQN. The approach is evaluated against two challenging scenarios of pedestrians crossing with occlusion and curved roads with a limited field of view. The algorithm is trained and evaluated using the SUMO traffic simulator. The proposed approach yields much better motion planning behavior compared to conventional RL algorithms and behaves comparably to humans driving style.
BibTeX
@inproceedings{rezaee2021motion,
  author = {Kasra Rezaee and Peyman Yadmellat and Simon Chamorro},
  title = {{Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=UrV1GRKypZ},
}
Figure