Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learning

Kasra Rezaee, Peyman Yadmellat, Simon Chamorro

TL;DR

Standard reinforcement learning (RL) agents are "optimists" that try to maximize average rewards—a dangerous strategy for self-driving cars where a single "below-average" event can be fatal. This paper introduces a Conservative RL framework that leverages distributional RL to optimize for the worst-case scenario rather than the mean. By focusing on the lower bound of potential outcomes, the agent learns human-like defensive driving behaviors, such as slowing down at blind corners and positioning itself for better visibility.

What Problem Does This Paper Address?

Motion planning for autonomous vehicles (AVs) is plagued by uncertainty, particularly from occlusions (e.g., a parked truck blocking a crosswalk or a sharp curve in a tunnel).

Traditional "classic" robotics approaches handle this by placing hypothetical "ghost" objects in every occluded area. While safe, this often leads to "frozen robot" syndrome—where the car is too terrified to move—and requires massive computational power to track every possibility. On the other hand, standard RL agents tend to ignore rare but catastrophic risks because they are trained to maximize the expected (average) return. If a blind corner is clear 99% of the time, a standard RL agent will likely speed through it, failing catastrophically the 1% of the time a pedestrian is actually there.

Methods

The authors propose Conservative Reinforcement Learning (CQR). The core innovation lies in how the agent perceives "value":

  1. Distributional RL: Instead of learning a single scalar value for a state-action pair (the "expected reward"), the authors use Quantile Regression. This allows the network to estimate the entire probability distribution of potential rewards.
  2. Optimizing the Lower Bound: Rather than taking the mean of this distribution, the authors modify the objective function to maximize the first quantile ($q_1$)—essentially the worst-case outcome.
  3. Algorithm Integration: They applied this "conservative" tweak to two popular RL architectures:
  4. CQR-DQN: A value-based approach.
  5. CQR-SAC (Soft Actor-Critic): A policy-based approach.
  6. Trajectory vs. Policy Evaluation: They compared two ways of looking ahead:
  7. $\tau$ (Trajectory): Assuming the vehicle follows a fixed path.
  8. $\pi$ (Policy): Assuming the vehicle can adapt its strategy in future steps. The policy-based evaluation ($\pi$) proved more flexible and effective.

Datasets

The study used the SUMO (Simulation of Urban MObility) traffic simulator to create two custom, high-stakes scenarios:

  • Pedestrian Crossing: A vehicle approaches a crosswalk where a large parked object blocks the view of pedestrians.
  • Curved Road: A sharp curve with a barrier blocking the line of sight, occasionally featuring a stalled vehicle (traffic jam) just out of view.

The authors used a curriculum learning approach, starting with easy visibility and gradually moving the occlusions closer to the road to "teach" the agent prudence.

Key Insights and Ideas

  • Safety via Pessimism: By forcing the agent to assume the "worst" is likely, it naturally adopts "defensive driving." It doesn't need to be told "slow down at blind spots"; it discovers that slowing down is the only way to keep the lower bound of its reward distribution high.
  • Emergent "Peeking" Behavior: One of the most striking results was that the conservative agents learned to maneuver to the left side of the lane when approaching a right-hand occlusion. This increases their field of view—a sophisticated behavior humans use but is rarely seen in standard RL agents.
  • The "Average Reward" Trap: The researchers found that looking at the average episode reward can be misleading. An algorithm might have a high average reward but a high collision rate. The conservative agents sometimes had lower average rewards (because they drive slower and more cautiously) but were significantly more successful at surviving.

Key Results

The conservative policy-based models (CQR-SAC$_\pi$ and CQR-DQN$_\pi$) outperformed both standard RL and naive rule-based baselines:

  • Collision Reduction: In the curved road scenario, standard SAC had a 37.56% collision rate, while the conservative CQR-SAC$_\pi$ slashed that to just 3.71%.
  • Human-Like Speed Profiles: The conservative agents slowed down significantly before blind spots (averaging ~4-5 m/s) compared to standard agents that maintained higher, riskier speeds (~10 m/s).
  • Robustness: While standard RL failed as the task complexity (occlusion proximity) increased, the conservative agents adapted their speed dynamically to maintain a "safety buffer."

Limitations and Future Work

  • Simulation Gap: The study was conducted entirely in SUMO. Real-world sensor noise and complex vehicle dynamics might present new challenges for the quantile estimation.
  • Static Environments: The current scenarios focus on static occlusions. The authors acknowledge that future work needs to include dynamic agents (other moving cars), where the ego-vehicle must infer the state of occluded areas by observing how other drivers are behaving.
  • Computational Overhead: While more efficient than some "hypothetical object" methods, distributional RL requires maintaining multiple quantiles, which increases the neural network's complexity.

Summary updated

Abstract

Motion planning under uncertainty is one of the main challenges in developing autonomous driving vehicles. In this work, we focus on the uncertainty in sensing and perception, resulted from a limited field of view, occlusions, and sensing range. This problem is often tackled by considering hypothetical hidden objects in occluded areas or beyond the sensing range to guarantee passive safety. However, this may result in conservative planning and expensive computation, particularly when numerous hypothetical objects need to be considered. We propose a reinforcement learning (RL) based solution to manage uncertainty by optimizing for the worst case outcome. This approach is in contrast to traditional RL, where the agents try to maximize the average expected reward. The proposed approach is built on top of the Distributional RL with its policy optimization maximizing the stochastic outcomes’ lower bound. This modification can be applied to a range of RL algorithms. As a proof-of-concept, the approach is applied to two different RL algorithms, Soft Actor-Critic and DQN. The approach is evaluated against two challenging scenarios of pedestrians crossing with occlusion and curved roads with a limited field of view. The algorithm is trained and evaluated using the SUMO traffic simulator. The proposed approach yields much better motion planning behavior compared to conventional RL algorithms and behaves comparably to humans driving style.
BibTeX
@inproceedings{rezaee2021motion,
  author = {Kasra Rezaee and Peyman Yadmellat and Simon Chamorro},
  title = {{Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=UrV1GRKypZ},
}

Loading related content...

Figure