Introduction
In the field of autonomous driving (AD), achieving high scores on standard benchmarks does not always translate to real-world safety. Currently, the nuPlan benchmark serves as the industry standard for closed-loop evaluation. However, researchers have observed a critical flaw: nuPlan is dominated by "basic" driving scenarios. Simple rule-based planners that merely follow the lane centerline (such as PDM-Closed) achieve near-perfect scores, yet they often fail when faced with obstacles or required lane changes.
To bridge this gap, a new study introduces interPlan, a closed-loop benchmark designed to stress-test planners on "long-tail" scenarios—rare, complex, and highly interactive events like navigating construction zones or accident sites. Alongside this benchmark, the authors propose HybridLLMPlanner, a system that leverages the reasoning capabilities of Large Language Models (LLMs) to handle these difficult edge cases.
The interPlan Benchmark
The researchers augmented existing nuPlan scenarios with 80 specific "edge case" elements. These are categorized into four primary types of challenges:
- Construction Zones: Requiring path deviations around cones and barriers.
- Accident Sites: Navigating around vehicles with non-standard orientations.
- Jaywalkers: Reacting to unpredictable pedestrian movements.
- Interactive Lane Changes: Merging into traffic with varying densities (Low, Medium, and High).
The benchmark reveals the "Centerline Trap": many state-of-the-art planners are essentially reactive centerline followers. When a lane is blocked in interPlan, these planners typically get stuck or collide because they lack the logic to leave their assigned lane safely.
HybridLLMPlanner: Combining Reasoning with Precision
To address the generalization gap, the team developed a two-stage hybrid architecture. This design separates high-level strategic decision-making from low-level geometric execution.
- Stage 1 (Behavior Planning): An LLM (such as Llama-2 or GPT-3.5) processes a natural language description of the scene. Using its inherent "common sense" world knowledge, it selects a high-level behavior, such as "overtake obstacle" or "stop and wait."
- Stage 2 (Motion Planning): A rule-based planner (based on PDM-Closed) receives this command and generates a precise, physically constrained trajectory.
This approach allows the system to "reason" about the environment. For instance, while a traditional pipeline might wait indefinitely for a crashed car to move, an LLM can identify the vehicle as part of an accident and decide to navigate around it.
Key Results and Insights
The experimental results highlight the extreme difficulty of the new benchmark. The top-performing rule-based planner, PDM-Closed, saw its score plummet from 92% on standard nuPlan (Val14) to just 42% on interPlan.
In contrast, the HybridLLMPlanner (using Llama-7B) established a new state-of-the-art on the benchmark with a score of 53%. Notably, these results were achieved in a zero-shot setting, meaning the LLM was not specifically fine-tuned on these scenarios. This suggests that the world models inherent in LLMs provide a significant advantage for generalization.
However, the study also identified a "toggling" problem: LLMs occasionally fluctuate rapidly between two behaviors (e.g., "overtake" vs. "wait"), leading to inconsistent vehicle movement. Furthermore, larger models did not always perform better; Llama-7B frequently outperformed Llama-13B, which occasionally made overly aggressive decisions resulting in collisions.
Conclusion
The interPlan benchmark exposes the limitations of current motion planners in handling the long tail of driving scenarios. While hybrid LLM-based systems show promise in providing the "common sense" necessary for complex navigation, significant work remains to stabilize their decision-making and synchronize their reasoning with real-time traffic dynamics. Future research will focus on fine-tuning LLMs with multimodal sensor data and auxiliary tasks to further improve safety and intent prediction.
Summary updated