Abstract
Reinforcement learning (RL), with its ability to explore and optimize policies in complex, dynamic decision-making tasks, has emerged as a promising approach to addressing motion planning (MoP) challenges in autonomous driving (AD). Despite rapid advancements in RL and AD, a systematic description and interpretation of the RL design process tailored to diverse driving tasks remains underdeveloped. This survey provides a comprehensive review of RL-based MoP for AD, focusing on lessons from task-specific perspectives. We first outline the fundamentals of RL methodologies, and then survey their applications in MoP, analyzing scenario-specific features and task requirements to shed light on their influence on RL design choices. Building on this analysis, we summarize key design experiences, extract insights from various driving task applications, and provide guidance for future implementations. Additionally, we examine the frontier challenges in RL-based MoP, review recent efforts to address these challenges, and propose strategies for overcoming unresolved issues.
Introduction: The Need for a Task-First Perspective
Reinforcement Learning has shown immense potential for solving complex, dynamic decision-making tasks in autonomous driving (AD). However, applying RL to AD is not a "one-size-fits-all" endeavor. A motion planning (MoP) model designed for simple highway car-following will likely fail in a chaotic urban intersection or an off-road environment.
As shown in the publication trends above, while research in both RL and AD has surged over the last decade, there is a notable gap in literature that explains why certain design choices are made for specific driving tasks. Most existing surveys categorize research by algorithm (e.g., grouping all PPO-based papers), but they often neglect the "how-to" of RL design tailored to diverse driving requirements. This paper addresses this gap by providing a systematic interpretation of the design process from a driving task perspective.
Methodology and Survey Structure
The survey follows a structured taxonomy to map the landscape of RL-based MoP across three primary dimensions: Policy Generation (Value-based vs. Actor-Critic), Agent Configuration (Single-agent vs. Hierarchical or Multi-agent), and Learning Mode (Online vs. Offline).
The core of the review analyzes how these methodological choices are applied to specific tasks, ranging from basic car-following to complex urban navigation and high-speed racing. By examining these scenarios, the survey extracts generic design paradigms and customized adaptations for various driving challenges.
Key Insights in RL Design for AD
The "Safety Filter" Paradigm
A major shift is occurring from "soft" safety (penalizing collisions in the reward function) to "hard" safety. Researchers are increasingly implementing Control Barrier Functions (CBFs) or Model Predictive Control (MPC) as a safety layer. These filters mask or correct unsafe RL actions before they reach the vehicle's actuators, ensuring physical constraints are met regardless of the RL agent's exploration.
Observation Hierarchy and Representations
Transitioning from raw sensor data to "Auxiliary Representations" has become a key trend. By feeding agents processed data like Risk Potential Fields or Occupancy Grid Maps, designers can significantly speed up training convergence and improve the agent's understanding of spatial relationships.
Task Differentiation via Hierarchical RL (HRL)
For complex tasks like urban navigation, Hierarchical RL is becoming the standard. This approach mimics human cognitive layers: a "high-level" agent makes semantic decisions (e.g., "change lane"), while a "low-level" agent handles continuous control (steering and acceleration).
LLMs as Feature Extractors
A frontier development involves using Large Language Models (LLMs) to translate multi-modal sensor data into task-relevant state representations. This helps agents incorporate "common sense" rules of the road that are difficult to encode mathematically.
Key Results and Performance Milestones
- Racing Dominance: RL has reached a milestone where it can surpass top human players in high-fidelity racing simulators like Gran Turismo.
- Generalization through Meta-RL: Meta-RL approaches allow agents to adapt to new traffic densities or weather conditions significantly faster than traditional RL by learning "how to learn" across multiple tasks.
- Efficiency via Learning from Demonstration (LfD): Using human driving data to initialize RL agents has proven effective in overcoming the "sparse reward" problem in complex scenarios like intersections.
Frontier Challenges and Future Work
Despite these successes, several challenges remain:
- The Sim-to-Real Gap: Real-world deployment is hindered by the lack of "closed-loop" interaction models in current data-driven simulators.
- Causal Reasoning: Current RL agents often act as "black boxes" reacting to correlations. Future work must integrate Causal Models so agents understand the underlying reasons for events, improving performance in rare "long-tail" edge cases.
- Safety vs. Utility Trade-off: Finding the balance between an overly conservative agent and a risky one remains a primary technical hurdle.
- LLM Hallucinations: While LLMs aid scene understanding, their potential to provide inaccurate interpretations of multi-modal inputs poses a significant risk for safety-critical motion planning.
Summary updated