Abstract
We introduce DriveAgent, a novel multi-agent autonomous driving framework that leverages large language model (LLM) reasoning combined with multimodal sensor fusion to enhance situational understanding and decision-making. DriveAgent uniquely integrates diverse sensor modalities—including camera, LiDAR, GPS, and IMU—with LLM-driven analytical processes structured across specialized agents. The framework operates through a modular agent-based pipeline comprising four principal modules: (i) a descriptive analysis agent identifying critical sensor data events based on filtered timestamps, (ii) dedicated vehicle-level analysis conducted by LiDAR and vision agents that collaboratively assess vehicle conditions and movements, (iii) environmental reasoning and causal analysis agents explaining contextual changes and their underlying mechanisms, and (iv) an urgency-aware decision-generation agent prioritizing insights and proposing timely maneuvers. This modular design empowers the LLM to effectively coordinate specialized perception and reasoning agents, delivering cohesive, interpretable insights into complex autonomous driving scenarios. Extensive experiments on challenging autonomous driving datasets demonstrate that DriveAgent is achieving superior performance on multiple metrics against baseline methods. These results validate the efficacy of the proposed LLM-driven multi-agent sensor fusion framework, underscoring its potential to substantially enhance the robustness and reliability of autonomous driving systems.
1. Introduction
Current autonomous driving (AD) systems generally fall into two categories: traditional modular pipelines and end-to-end neural networks. Both approaches often struggle to integrate heterogeneous sensor data effectively, particularly in "edge cases" where one sensor might be unreliable—for instance, when a camera is blurry or misaligned. While recent research has explored using Large Language Models (LLMs) for driving, these methods frequently rely on simple visual descriptions and lack the "common sense" reasoning required to resolve contradictions between sensors or understand the underlying causes of environmental changes.
DriveAgent bridges this gap by creating a system that uses LLM-driven structured reasoning to provide interpretable, robust situational awareness. By structuring reasoning into specialized agents, the framework performs cross-sensor "sanity checks" and causal analysis to understand why a scene is changing.
2. Methodology
DriveAgent operates through a pipeline of four specialized modules (M1–M4), each handled by distinct agents:
- M1: Descriptive Analysis: This module uses GPS and IMU data to filter for "critical timestamps"—moments where significant motion, such as acceleration or turning, occurs. This prevents the LLM from being overwhelmed by redundant data.
- M2: Vehicle Reasoning: In this "sanity check" phase, a LiDAR agent and a Vision agent independently describe the scene. An analyzer agent then compares these descriptions; if the Euclidean distance between a LiDAR-detected object and a camera-inferred object is too large, the system flags a potential sensor calibration error.
- M3: Environmental Reasoning: This module analyzes changes between consecutive timestamps. It utilizes a Causal Analysis agent to determine if an object is "self-moving" (e.g., a pedestrian) or "externally influenced," providing deeper context for decision-making.
- M4: Response Generation: A final agent aggregates all insights, scores them based on urgency (balancing safety and efficiency), and generates a prioritized, interpretable response.
The system uses a fine-tuned LLaMA-3.2-Vision (11B) as its backbone, optimized using Low-Rank Adaptation (LoRA) on structured JSON-formatted prompts.
3. Experimental Evaluation
The researchers introduced a new Three-Tier Driving Dataset collected from a real-world vehicle equipped with a Robosense Ruby-128 LiDAR and Basler cameras, ranging from controlled environments (R1) to challenging urban routes with obstructions (R3).
3.1 Key Results
- Detection Accuracy: DriveAgent achieved a Precision of 89.96% and an F1-score of 71.62, outperforming GPT-4o (80.98% Precision) and Claude-3.7-Sonnet (83.83% Precision).
- Robustness to Sensor Failure: In tasks where camera views were intentionally misaligned, DriveAgent maintained 96.84% accuracy on the R2 route. In contrast, zero-shot and Chain-of-Thought (CoT) baselines saw their accuracy plummet because they could not resolve the sensor conflict.
- Causal Understanding: In identifying independently moving objects, DriveAgent reached 58.95% accuracy, a significant improvement over the 37.89% achieved by standard zero-shot LLM approaches.
4. Discussion and Conclusion
One of the most significant insights from DriveAgent is its ability to detect its own sensor failures. By comparing 3D coordinates from LiDAR with 2D-to-3D projections from the camera, the system can explicitly identify misalignment rather than making an incorrect driving decision. Furthermore, the transition from purely descriptive modeling to causal reasoning is critical for predicting future environmental states.
While the multi-agent approach provides excellent interpretability, future work is needed to address the reasoning overhead and latency implications of multiple LLM inference passes in real-time environments. Additionally, expanding the dataset to include hazardous weather conditions like fog or heavy rain will further test the limits of this multimodal sensor fusion framework.
Summary updated