Abstract
Artificial intelligence (AI) plays a crucial role in autonomous driving (AD) research, propelling its development towards intelligence and efficiency. Currently, the development of AD technology follows two main technical paths: modularization and end-to-end. Modularization decomposes the driving task into modules such as perception, prediction, planning, and control, and trains them separately. Due to the inconsistency of training objectives between modules, the integrated effect suffers from bias. End-to-end attempts to address this issue by utilizing a single model that directly maps from sensor data to control signals. This path has limited learning capabilities in a comprehensive set of features and struggles to handle unpredictable long-tail events and complex urban traffic scenarios. In light of the challenges faced by both paths, many researchers believe that Large Language Models (LLMs), with their powerful reasoning abilities and extensive knowledge, could offer a solution. This paper conducts a thorough analysis of the potential applications of LLMs in AD systems, including exploring their optimization strategies in both modular and end-to-end approaches, with a particular focus on how LLMs can tackle the problems and challenges present in current solutions. Furthermore, we discuss whether LLM-based artificial general intelligence (AGI) can be a key to achieving high-level AD and analyze the potential limitations and challenges that LLMs may encounter.
The Architectural Divide in Autonomous Driving
Autonomous driving research has historically converged on two primary paradigms: Modularization and End-to-End systems. Modular systems break the driving task into discrete components—perception, prediction, planning, and control—which are developed and optimized independently. While this offers interpretability, it often leads to information loss and conflicting objectives between modules. Conversely, End-to-End systems attempt to map raw sensor data directly to control signals using a single neural network. While this reduces complexity, these "black box" models often lack the reasoning depth required for complex urban environments.
Both paths face a common "glass ceiling": the inability to handle "long-tail" events—rare, unpredictable scenarios that are underrepresented in training data. LLMs are emerging as a potential solution by shifting the paradigm from a purely data-driven approach to a knowledge-driven one, infusing systems with human-like common sense and reasoning.
Integration Strategies for LLMs
The integration of LLMs into the AD pipeline generally follows three main directions:
- Modular Enhancement: LLMs or Vision-Language Models (VLMs) are used to improve specific tasks. For instance, they can enhance scene understanding in perception or generate high-level behavioral "meta-actions" (e.g., "slow down for the puddle") that traditional controllers then execute.
- LLMs as End-to-End Agents: Here, the LLM serves as the central "brain," receiving multimodal inputs (video, LiDAR, text) and directly outputting driving plans or control signals.
- Data and Simulation: LLMs are utilized to generate synthetic training data, annotate complex 3D scenes with natural language, and create realistic simulation scenarios for testing.
Data Fundamentals and Benchmarks
A new generation of datasets is supporting this shift toward LLM-based driving. These include:
- LLM-Assisted Annotation: Datasets like Reason2Drive and LingoQA provide natural language explanations for driving scenes, helping models understand the "why" behind driving actions.
- Synthetic Generation: Tools like WEDGE use generative models to create synthetic images of rare weather conditions to bolster model robustness.
- Closed-Loop Benchmarks: DriveLM and LMDrive focus on following natural language instructions within simulations, testing the model's ability to act on complex commands.
Key Insights: From Seeing to Understanding
The most significant contribution of LLMs is the transition from simple pattern recognition to semantic understanding. LLMs bring "common sense"—the distilled human experience that allows a system to infer, for example, that a ball rolling into the street likely precedes a child.
Furthermore, LLMs improve interpretability. Unlike traditional deep learning models, an LLM-powered system can provide a "Chain of Thought" (CoT) explanation for its actions. If a vehicle stops abruptly, the system can articulate its reasoning (e.g., "Detected an obscured pedestrian near the crosswalk"), which is vital for safety validation and human trust.
Performance Against Core Challenges
The survey mapped LLM capabilities against ten core AD challenges, yielding several key results:
- High Impact: LLMs excel at perception in uncontrolled environments, establishing unified data annotation standards, and integrating disparate planning tasks.
- Moderate Impact: They show promise in handling complex kinematic constraints and improving training efficiency, though these areas remain active research fronts.
- Low Impact: Currently, LLMs struggle with low-level, high-frequency control requirements due to significant computational latency.
Limitations and Future Outlook
Despite their potential, LLMs are not yet a total panacea. Several "deal-breakers" remain:
- Hallucination: In safety-critical driving, a hallucinated clear path can lead to catastrophic failure.
- Real-time Latency: The computational weight of LLMs often results in response times too slow for split-second emergency maneuvers.
- Physical Grounding: LLMs lack an intuitive, embodied understanding of physical forces like momentum and friction.
The future likely belongs to a hybrid model. In this architecture, mature, low-latency modules handle immediate safety and control, while LLMs provide the high-level reasoning and common sense necessary for navigating the complexities of the real world.
Summary updated