Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving

Guizhe Jin, Zhuoren Li, Bo Leng, Ran Yu, Lu Xiong, Chen SunIEEE RA-L 2025 · DOI · OpenReview

Abstract

Reinforcement Learning (RL) is increasingly used in autonomous driving (AD) and shows clear advantages. However, most RL-based AD methods overlook policy structure design. An RL policy that only outputs short-timescale vehicle control commands results in fluctuating driving behavior due to fluctuations in network outputs, while one that only outputs long-timescale driving goals cannot achieve unified optimality of driving behavior and control. Therefore, we propose a multi-timescale hierarchical reinforcement learning approach. Our approach adopts a hierarchical policy structure, where high- and low-level RL policies are unified-trained to produce long-timescale motion guidance and short-timescale control commands, respectively. Therein, motion guidance is explicitly represented by hybrid actions to capture multimodal driving behaviors on structured road and support incremental low-level extend-state updates. Additionally, a hierarchical safety mechanism is designed to ensure multi-timescale safety. Evaluation in simulator-based and HighD dataset-based highway multi-lane scenarios demonstrates that our approach significantly improves AD performance, effectively increasing driving efficiency, action consistency and safety.

Introduction

Autonomous driving (AD) systems powered by Reinforcement Learning (RL) often struggle with a fundamental trade-off in policy frequency. High-frequency policies that directly output control commands (e.g., steering and acceleration) frequently exhibit jittery, inconsistent behavior due to network output fluctuations. Conversely, low-frequency policies that only set long-term goals lack the fine-grained control necessary to optimize for complex, dynamic traffic environments. Humans solve this by driving hierarchically: planning a path over a long timescale while subconsciously managing vehicle controls over a short timescale. This paper introduces a Multi-Timescale Hierarchical Reinforcement Learning (MTHRL) framework to replicate this natural hierarchy, bridging the gap between strategic behavior and precise control.

Methodology

The proposed MTHRL framework consists of two jointly-trained RL policies operating at different frequencies to balance strategic planning and immediate execution.

The MTHRL framework architecture, showing the interaction between high-level policy, low-level policy, and the safety mechanism.

Hierarchical Policy Structure

  • High-Level Policy ($\pi^h$): Operates at a 1.0s interval. It outputs Motion Guidance using a hybrid action space: discrete actions for lane selection and continuous actions for longitudinal target positions. This guidance is mathematically represented as a polynomial path.
  • Low-Level Policy ($\pi^l$): Operates at a 0.1s interval. It receives the high-level guidance as part of its extended state and outputs specific control commands (steering angle and acceleration).

Incremental Updates and Safety

A key innovation is the Incremental Guidance Update mechanism. As the vehicle progresses, the high-level target is updated at every low-level step to maintain physical consistency. Furthermore, a Hierarchical Safety Mechanism based on Artificial Potential Fields (APF) evaluates risk at both timescales. This module can override RL outputs with safer, conservative actions if a collision risk is detected, ensuring safety throughout the training process.

Experimental Evaluation

The framework was validated in two distinct environments to test both tactical decision-making and real-world applicability:

  1. Highway-Env: A simulation environment for multi-lane highway traffic.
  2. HighD Dataset: A large-scale real-world dataset of vehicle trajectories recorded on German highways, providing a benchmark for realistic traffic density.

Results and Discussion

The proposed method, MTHRL-HS, significantly outperformed standard PPO and other hierarchical RL variants across several key metrics.

Training performance comparison showing total reward across episodes for various RL methods in (a) simulator and (b) real-world dataset scenarios.
  • Efficiency and Performance: As shown in the training curves, MTHRL-HS achieves higher total rewards more quickly than baseline methods. In testing, it improved driving speed by 13.1% over suboptimal hierarchical approaches.
  • Consistency: By decoupling high-level planning from low-level execution, the system reduced steering and acceleration fluctuations by approximately 50%, resulting in smoother driving behavior.
  • Safety: The hierarchical safety layer proved highly effective. In the Highway-Env simulator, the collision rate was reduced to 0.03%, while on the real-world HighD dataset, it reached 0.02%, surpassing all tested baselines.

Conclusion

The MTHRL framework successfully unifies high-level behavior planning with low-level vehicle control. By employing a multi-timescale structure and an incremental update mechanism, the system achieves superior efficiency, safety, and consistency. Future work will focus on extending this approach to more complex urban environments, including intersections and interactions with pedestrians, while incorporating higher-fidelity vehicle dynamics models.

Summary updated

Abstract

Reinforcement Learning (RL) is increasingly used in autonomous driving (AD) and shows clear advantages. However, most RL-based AD methods overlook policy structure design. An RL policy that only outputs short-timescale vehicle control commands results in fluctuating driving behavior due to fluctuations in network outputs, while one that only outputs long-timescale driving goals cannot achieve unified optimality of driving behavior and control. Therefore, we propose a multi-timescale hierarchical reinforcement learning approach. Our approach adopts a hierarchical policy structure, where high- and low-level RL policies are unified-trained to produce long-timescale motion guidance and short-timescale control commands, respectively. Therein, motion guidance is explicitly represented by hybrid actions to capture multimodal driving behaviors on structured road and support incremental low-level extend-state updates. Additionally, a hierarchical safety mechanism is designed to ensure multi-timescale safety. Evaluation in simulator-based and HighD dataset-based highway multi-lane scenarios demonstrates that our approach significantly improves AD performance, effectively increasing driving efficiency, action consistency and safety.
BibTeX
@article{jin2025multi,
  author = {Guizhe Jin and Zhuoren Li and Bo Leng and Ran Yu and Lu Xiong and Chen Sun},
  title = {{Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  doi = {10.1109/lra.2025.3623016},
}
Figure