A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning

Jinning Li, Liting Sun, Jianyu Chen, Masayoshi Tomizuka, Wei Zhan

Introduction

Autonomous vehicles (AVs) face a fundamental "Goldilocks" problem in behavior planning. Classical optimization or sampling-based planners are safe and interpretable but often struggle with real-time performance in interactive environments and are notoriously difficult to tune across diverse scenarios. Conversely, end-to-end Reinforcement Learning (RL) can learn complex, adaptive behaviors but operates as a "black box" without formal safety guarantees—a critical barrier for real-world deployment.

This paper introduces H-CtRL, a hierarchical framework designed to bridge this gap by combining the high-level decision-making power of RL with the rigorous safety constraints of classical motion controllers.

The H-CtRL Framework

The H-CtRL (Hierarchical behavior planning with low-level safe Controllers and high-level Reinforcement Learning) framework decouples the planning task into two distinct layers.

The H-CtRL hierarchical architecture combining high-level RL decision-making with low-level safe controllers.

High-Level Coordinator

A Double Deep Q-Network (DDQN) acts as the system's brain. Rather than outputting raw control commands like steering or acceleration, the RL agent selects from a discrete set of $n$ low-level controllers. Decisions are made every $p$ timesteps (e.g., every 1 second) to maintain behavioral consistency and reduce the complexity of the action space, making training more stable and sample-efficient.

Low-Level Safe Controllers

The execution layer consists of multiple Constrained Iterative Linear Quadratic Regulators (CILQR). While each controller shares the same hard safety and dynamic constraints, they are tuned with different "personalities" based on reference speeds (ranging from 0 m/s to 9 m/s). Because the CILQR solves an optimization problem with hard constraints, any action sent to the environment is guaranteed to be safe within the planning horizon. The RL's role is simply to choose the most appropriate "mode" (e.g., aggressive vs. conservative) for the current traffic context.

Experimental Setup

The researchers evaluated H-CtRL using a custom simulator built on the INTERACTION Dataset, which provides high-fidelity, real-world traffic data. Testing focused on two high-interactivity scenarios: a complex intersection requiring unprotected turns and a large roundabout.

Complex driving scenarios from the INTERACTION dataset used for evaluation: a busy intersection (left) and a roundabout (right).

Key Results and Insights

The H-CtRL framework was compared against fixed-parameter CILQR baselines (both conservative and aggressive) and a random switching agent.

  • Success and Safety: In the intersection scenario, H-CtRL achieved an 85% completion rate, significantly outperforming the conservative planner (24% success due to timeouts) and the aggressive planner (9% success due to crashes). In the roundabout, it maintained a low collision rate (0.08), nearly matching the safest classical baseline (0.05) while being far more efficient.
  • Context-Aware Aggression: Visualization of the agent's behavior revealed that it successfully learned "social norms." It utilizes high-speed controllers to enter intersections, switches to low-speed/waiting modes when oncoming traffic is detected, and accelerates quickly to exit.
  • Safety as a Constraint: Unlike many RL approaches that use negative rewards to discourage collisions, H-CtRL treats safety as a hard constraint at the control level. The RL agent cannot choose an unsafe action; it can only choose a more aggressive controller that operates closer to the safety boundary.

Conclusion and Future Work

H-CtRL demonstrates that hierarchical structures can provide the adaptability needed for complex urban driving without sacrificing safety. However, the authors noted that the agent occasionally exhibits abrupt braking when following lead vehicles, as the current CILQR prioritizes safety over passenger comfort (jerk). Future research will focus on integrating a wider variety of low-level planners and incorporating optimization for passenger comfort into the control layer.

Summary updated

Abstract

Autonomous vehicles need to handle various traffic conditions and make safe and efficient decisions and maneuvers. However, on the one hand, a single optimization/sampling-based motion planner cannot efficiently generate safe trajectories in real time, particularly when there are many interactive vehicles near by. On the other hand, end-to-end learning methods cannot assure the safety of the outcomes. To address this challenge, we propose a hierarchical behavior planning framework with a set of low-level safe controllers and a high-level reinforcement learning algorithm (H-CtRL) as a coordinator for the low-level controllers. Safety is guaranteed by the low-level optimization/sampling-based controllers, while the high-level reinforcement learning algorithm makes H-CtRL an adaptive and efficient behavior planner. To train and test our proposed algorithm, we built a simulator that can reproduce traffic scenes using real-world datasets. The proposed HCtRL is proved to be effective in various realistic simulation scenarios, with satisfying performance in terms of both safety and efficiency.
BibTeX
@inproceedings{li2021safe,
  author = {Jinning Li and Liting Sun and Jianyu Chen and Masayoshi Tomizuka and Wei Zhan},
  title = {{A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=sLjZeBLwKb},
}
Figure