Hierarchical Reinforcement Learning for Autonomous Decision Making and Motion Planning of Intelligent Vehicles

Yang Lu, Xin Xu, Xinglong Zhang, Lilin Qian, Xing ZhouIEEE Access 2020 · OpenReview

Introduction

Autonomous vehicles (AVs) have made significant strides in structured environments like highways. However, "unprotected" maneuvers—such as turning left against oncoming traffic or merging into high-speed lanes from a ramp—remain significant hurdles. These scenarios require complex interactions with other road users and precise control of vehicle dynamics. A common failure point in current AV stacks is the disconnect between high-level behavioral decisions and low-level motion execution. If the "brain" (decision-making) commands a maneuver that the "muscles" (motion planning and control) cannot physically achieve due to tire grip or suspension limits, safety is compromised.

To address this, researchers from the National University of Defense Technology (NUDT) have developed a hierarchical reinforcement learning (HRL) framework. This system ensures that high-level strategic decisions are inherently feasible for the vehicle's physical capabilities, bridging the gap between strategy and execution.

Proposed Framework

The proposed approach utilizes a two-layer architecture designed to decouple strategic decision-making from tactical motion planning while maintaining physical consistency.

Overview of the proposed hierarchical reinforcement learning framework, illustrating the offline training and real-time testing processes.

Higher Layer: Decision Making with USP-KLSPI

The higher layer is responsible for behavioral decisions, such as selecting target speeds and lane-change timings. It employs a novel algorithm: Uneven Sampling and Pooling Kernel-based Least-Squares Policy Iteration (USP-KLSPI).

  1. Uneven Sampling: Traditional RL often suffers from sample inefficiency. USP-KLSPI mimics human attention by sampling more densely in high-risk intervals (e.g., when other vehicles are close) and less frequently in empty spaces. This focuses the learning process on critical interactions.
  2. Pooling Strategy: To manage the high-dimensional state space of traffic, a max-pooling approach compresses data into representative features, allowing the agent to generalize across varying numbers of surrounding vehicles.
  3. High-Fidelity Dynamics: Crucially, decisions are evaluated against a 14-degree-of-freedom (DOF) vehicle model. This model accounts for complex dynamics like tire-road interaction and suspension movement, ensuring that the higher layer only selects maneuvers that the vehicle can actually perform.

Lower Layer: Motion Planning with DHP

The lower layer translates high-level decisions into specific control inputs. It uses Dual Heuristic Programming (DHP) to solve the lateral motion-planning problem (steering). The longitudinal profile (speed) is inherited directly from the higher layer. By training the DHP agent in a batch-mode manner, the system achieves high computational efficiency, making it suitable for real-time deployment.

Experimental Evaluation

The framework was tested in a high-fidelity simulation environment across two challenging scenarios:

  • Scenario A (Unprotected Left Turn): Navigating an intersection without traffic signals against oncoming traffic.
  • Scenario B (Multi-lane Merging): Entering a three-lane highway from a side ramp with dense traffic.

Key Results

The USP-KLSPI method demonstrated superior performance compared to standard KLSPI and rule-based "Expert" policies:

  • Efficiency: In the merging scenario, USP-KLSPI converged in 1401 seconds, roughly 10% faster than standard KLSPI, validating the effectiveness of the uneven sampling strategy.
  • Safety and Success: In the left-turn scenario, the HRL framework achieved a 99.6% completion rate, significantly higher than the Expert Policy's 82.2%.
  • Smoothness: The "Obstacle Avoidance Assistance Rate" (OAAR)—which measures how often a safety backup must override the planner—was only 0.7% for the proposed system, compared to 30.1% for the expert policy. This indicates that the HRL system makes more proactive, safer decisions that rarely require emergency intervention.

Conclusion and Future Directions

The hierarchical approach successfully integrates high-level strategy with low-level physical constraints. By embedding vehicle dynamics into the decision-making process and optimizing sampling strategies, the researchers have created a more robust and efficient AV control system.

Future work will focus on incorporating more complex traffic rules and passenger comfort metrics. Additionally, while the 14-DOF simulation is highly accurate, the ultimate validation will come from transitioning the framework to real-world vehicle testing on physical tracks.

Summary updated

Abstract

Autonomous decision making and motion planning in complex dynamic traffic environments, such as left-turn without traffic signals and multi-lane merging from side-ways, are still challenging tasks for intelligent vehicles. It is difficult to generate optimized behavior decisions while considering the motion capabilities and dynamic properties of intelligent vehicles. Aiming at the above problems, this article proposes a hierarchical reinforcement learning approach for autonomous decision making and motion planning in complex dynamic traffic scenarios. The proposed approach consists of two layers. At the higher layer, a kernel-based least-squares policy iteration algorithm with uneven sampling and pooling strategy (USP-KLSPI) is presented for solving the decision-making problems. The motion capabilities of the ego vehicle and the surrounding vehicles are evaluated with a high-fidelity dynamic model in the decision-making layer. By doing so, the consistency between the decisions generated at the higher layer and the operations in the lower planning layer can be well guaranteed. The lower layer addresses the motion-planning problem in the lateral direction using a dual heuristic programming (DHP) algorithm learned in a batch-mode manner, while the velocity profile in the longitudinal direction is inherited from the higher layer. Extensive simulations are conducted in complex traffic conditions including left-turn without traffic signals and multi-lane merging from side-ways scenarios. The results demonstrate the effectiveness and efficiency of the proposed approach in realizing optimized decision making and motion planning in complex environments.
BibTeX
@inproceedings{lu2020hierarchical,
  author = {Yang Lu and Xin Xu and Xinglong Zhang and Lilin Qian and Xing Zhou},
  title = {{Hierarchical Reinforcement Learning for Autonomous Decision Making and Motion Planning of Intelligent Vehicles}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=GBliqMz26pG},
}
Figure