Introduction
Autonomous vehicles (AVs) face a fundamental "Goldilocks" problem in behavior planning. Classical optimization or sampling-based planners are safe and interpretable but often struggle with real-time performance in interactive environments and are notoriously difficult to tune across diverse scenarios. Conversely, end-to-end Reinforcement Learning (RL) can learn complex, adaptive behaviors but operates as a "black box" without formal safety guarantees—a critical barrier for real-world deployment.
This paper introduces H-CtRL, a hierarchical framework designed to bridge this gap by combining the high-level decision-making power of RL with the rigorous safety constraints of classical motion controllers.
The H-CtRL Framework
The H-CtRL (Hierarchical behavior planning with low-level safe Controllers and high-level Reinforcement Learning) framework decouples the planning task into two distinct layers.
High-Level Coordinator
A Double Deep Q-Network (DDQN) acts as the system's brain. Rather than outputting raw control commands like steering or acceleration, the RL agent selects from a discrete set of $n$ low-level controllers. Decisions are made every $p$ timesteps (e.g., every 1 second) to maintain behavioral consistency and reduce the complexity of the action space, making training more stable and sample-efficient.
Low-Level Safe Controllers
The execution layer consists of multiple Constrained Iterative Linear Quadratic Regulators (CILQR). While each controller shares the same hard safety and dynamic constraints, they are tuned with different "personalities" based on reference speeds (ranging from 0 m/s to 9 m/s). Because the CILQR solves an optimization problem with hard constraints, any action sent to the environment is guaranteed to be safe within the planning horizon. The RL's role is simply to choose the most appropriate "mode" (e.g., aggressive vs. conservative) for the current traffic context.
Experimental Setup
The researchers evaluated H-CtRL using a custom simulator built on the INTERACTION Dataset, which provides high-fidelity, real-world traffic data. Testing focused on two high-interactivity scenarios: a complex intersection requiring unprotected turns and a large roundabout.
Key Results and Insights
The H-CtRL framework was compared against fixed-parameter CILQR baselines (both conservative and aggressive) and a random switching agent.
- Success and Safety: In the intersection scenario, H-CtRL achieved an 85% completion rate, significantly outperforming the conservative planner (24% success due to timeouts) and the aggressive planner (9% success due to crashes). In the roundabout, it maintained a low collision rate (0.08), nearly matching the safest classical baseline (0.05) while being far more efficient.
- Context-Aware Aggression: Visualization of the agent's behavior revealed that it successfully learned "social norms." It utilizes high-speed controllers to enter intersections, switches to low-speed/waiting modes when oncoming traffic is detected, and accelerates quickly to exit.
- Safety as a Constraint: Unlike many RL approaches that use negative rewards to discourage collisions, H-CtRL treats safety as a hard constraint at the control level. The RL agent cannot choose an unsafe action; it can only choose a more aggressive controller that operates closer to the safety boundary.
Conclusion and Future Work
H-CtRL demonstrates that hierarchical structures can provide the adaptability needed for complex urban driving without sacrificing safety. However, the authors noted that the agent occasionally exhibits abrupt braking when following lead vehicles, as the current CILQR prioritizes safety over passenger comfort (jerk). Future research will focus on integrating a wider variety of low-level planners and incorporating optimization for passenger comfort into the control layer.
Summary updated