A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective

Zhuoren Li, Gang Jin, Ran Yu, Zhiwen Chen, Nan Li, Wei Han, Xiong Lu, Bo Leng, Jia Hu, Ilya Kolmanovsky, Dimitar FilevarXiv 2025 · DOI · arXiv

Abstract

Reinforcement learning (RL), with its ability to explore and optimize policies in complex, dynamic decision-making tasks, has emerged as a promising approach to addressing motion planning (MoP) challenges in autonomous driving (AD). Despite rapid advancements in RL and AD, a systematic description and interpretation of the RL design process tailored to diverse driving tasks remains underdeveloped. This survey provides a comprehensive review of RL-based MoP for AD, focusing on lessons from task-specific perspectives. We first outline the fundamentals of RL methodologies, and then survey their applications in MoP, analyzing scenario-specific features and task requirements to shed light on their influence on RL design choices. Building on this analysis, we summarize key design experiences, extract insights from various driving task applications, and provide guidance for future implementations. Additionally, we examine the frontier challenges in RL-based MoP, review recent efforts to address these challenges, and propose strategies for overcoming unresolved issues.

Introduction: The Need for a Task-First Perspective

Reinforcement Learning has shown immense potential for solving complex, dynamic decision-making tasks in autonomous driving (AD). However, applying RL to AD is not a "one-size-fits-all" endeavor. A motion planning (MoP) model designed for simple highway car-following will likely fail in a chaotic urban intersection or an off-road environment.

Publication trends showing the growth of RL and AD research alongside the relative scarcity of specific RL-based AD surveys.

As shown in the publication trends above, while research in both RL and AD has surged over the last decade, there is a notable gap in literature that explains why certain design choices are made for specific driving tasks. Most existing surveys categorize research by algorithm (e.g., grouping all PPO-based papers), but they often neglect the "how-to" of RL design tailored to diverse driving requirements. This paper addresses this gap by providing a systematic interpretation of the design process from a driving task perspective.

Methodology and Survey Structure

The survey follows a structured taxonomy to map the landscape of RL-based MoP across three primary dimensions: Policy Generation (Value-based vs. Actor-Critic), Agent Configuration (Single-agent vs. Hierarchical or Multi-agent), and Learning Mode (Online vs. Offline).

The logical flow of the survey, moving from RL basics to task-specific reviews and design lessons.

The core of the review analyzes how these methodological choices are applied to specific tasks, ranging from basic car-following to complex urban navigation and high-speed racing. By examining these scenarios, the survey extracts generic design paradigms and customized adaptations for various driving challenges.

Key Insights in RL Design for AD

The "Safety Filter" Paradigm

A major shift is occurring from "soft" safety (penalizing collisions in the reward function) to "hard" safety. Researchers are increasingly implementing Control Barrier Functions (CBFs) or Model Predictive Control (MPC) as a safety layer. These filters mask or correct unsafe RL actions before they reach the vehicle's actuators, ensuring physical constraints are met regardless of the RL agent's exploration.

Observation Hierarchy and Representations

Transitioning from raw sensor data to "Auxiliary Representations" has become a key trend. By feeding agents processed data like Risk Potential Fields or Occupancy Grid Maps, designers can significantly speed up training convergence and improve the agent's understanding of spatial relationships.

Task Differentiation via Hierarchical RL (HRL)

For complex tasks like urban navigation, Hierarchical RL is becoming the standard. This approach mimics human cognitive layers: a "high-level" agent makes semantic decisions (e.g., "change lane"), while a "low-level" agent handles continuous control (steering and acceleration).

LLMs as Feature Extractors

A frontier development involves using Large Language Models (LLMs) to translate multi-modal sensor data into task-relevant state representations. This helps agents incorporate "common sense" rules of the road that are difficult to encode mathematically.

Key Results and Performance Milestones

  • Racing Dominance: RL has reached a milestone where it can surpass top human players in high-fidelity racing simulators like Gran Turismo.
  • Generalization through Meta-RL: Meta-RL approaches allow agents to adapt to new traffic densities or weather conditions significantly faster than traditional RL by learning "how to learn" across multiple tasks.
  • Efficiency via Learning from Demonstration (LfD): Using human driving data to initialize RL agents has proven effective in overcoming the "sparse reward" problem in complex scenarios like intersections.

Frontier Challenges and Future Work

Despite these successes, several challenges remain:

  • The Sim-to-Real Gap: Real-world deployment is hindered by the lack of "closed-loop" interaction models in current data-driven simulators.
  • Causal Reasoning: Current RL agents often act as "black boxes" reacting to correlations. Future work must integrate Causal Models so agents understand the underlying reasons for events, improving performance in rare "long-tail" edge cases.
  • Safety vs. Utility Trade-off: Finding the balance between an overly conservative agent and a risky one remains a primary technical hurdle.
  • LLM Hallucinations: While LLMs aid scene understanding, their potential to provide inaccurate interpretations of multi-modal inputs poses a significant risk for safety-critical motion planning.

Summary updated

Abstract

Reinforcement learning (RL), with its ability to explore and optimize policies in complex, dynamic decision-making tasks, has emerged as a promising approach to addressing motion planning (MoP) challenges in autonomous driving (AD). Despite rapid advancements in RL and AD, a systematic description and interpretation of the RL design process tailored to diverse driving tasks remains underdeveloped. This survey provides a comprehensive review of RL-based MoP for AD, focusing on lessons from task-specific perspectives. We first outline the fundamentals of RL methodologies, and then survey their applications in MoP, analyzing scenario-specific features and task requirements to shed light on their influence on RL design choices. Building on this analysis, we summarize key design experiences, extract insights from various driving task applications, and provide guidance for future implementations. Additionally, we examine the frontier challenges in RL-based MoP, review recent efforts to addresse these challenges, and propose strategies for overcoming unresolved issues.
BibTeX
@misc{li2025survey,
  author = {Zhuoren Li and Gang Jin and Ran Yu and Zhiwen Chen and Nan Li and Wei Han and Xiong Lu and Bo Leng and Jia Hu and Ilya Kolmanovsky and Dimitar Filev},
  title = {{A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective}},
  year = {2025},
  eprint = {2503.23650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.23650},
  howpublished = {arXiv},
}
Figure