Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?

Marcel Hallgarten, Julián Zapata, Martin Stoll, Katrin Renz, Andreas Zell

Introduction

In the field of autonomous driving (AD), achieving high scores on standard benchmarks does not always translate to real-world safety. Currently, the nuPlan benchmark serves as the industry standard for closed-loop evaluation. However, researchers have observed a critical flaw: nuPlan is dominated by "basic" driving scenarios. Simple rule-based planners that merely follow the lane centerline (such as PDM-Closed) achieve near-perfect scores, yet they often fail when faced with obstacles or required lane changes.

To bridge this gap, a new study introduces interPlan, a closed-loop benchmark designed to stress-test planners on "long-tail" scenarios—rare, complex, and highly interactive events like navigating construction zones or accident sites. Alongside this benchmark, the authors propose HybridLLMPlanner, a system that leverages the reasoning capabilities of Large Language Models (LLMs) to handle these difficult edge cases.

The interPlan Benchmark

The researchers augmented existing nuPlan scenarios with 80 specific "edge case" elements. These are categorized into four primary types of challenges:

  1. Construction Zones: Requiring path deviations around cones and barriers.
  2. Accident Sites: Navigating around vehicles with non-standard orientations.
  3. Jaywalkers: Reacting to unpredictable pedestrian movements.
  4. Interactive Lane Changes: Merging into traffic with varying densities (Low, Medium, and High).
Figure 3: Examples of long-tail scenarios in the interPlan benchmark, including construction zones with barriers and accident sites with stalled vehicles.

The benchmark reveals the "Centerline Trap": many state-of-the-art planners are essentially reactive centerline followers. When a lane is blocked in interPlan, these planners typically get stuck or collide because they lack the logic to leave their assigned lane safely.

HybridLLMPlanner: Combining Reasoning with Precision

To address the generalization gap, the team developed a two-stage hybrid architecture. This design separates high-level strategic decision-making from low-level geometric execution.

  • Stage 1 (Behavior Planning): An LLM (such as Llama-2 or GPT-3.5) processes a natural language description of the scene. Using its inherent "common sense" world knowledge, it selects a high-level behavior, such as "overtake obstacle" or "stop and wait."
  • Stage 2 (Motion Planning): A rule-based planner (based on PDM-Closed) receives this command and generates a precise, physically constrained trajectory.
Figure 1: Visualization of complex maneuvers required in interPlan, such as nudging around obstacles and performing lane changes in traffic.

This approach allows the system to "reason" about the environment. For instance, while a traditional pipeline might wait indefinitely for a crashed car to move, an LLM can identify the vehicle as part of an accident and decide to navigate around it.

Key Results and Insights

The experimental results highlight the extreme difficulty of the new benchmark. The top-performing rule-based planner, PDM-Closed, saw its score plummet from 92% on standard nuPlan (Val14) to just 42% on interPlan.

In contrast, the HybridLLMPlanner (using Llama-7B) established a new state-of-the-art on the benchmark with a score of 53%. Notably, these results were achieved in a zero-shot setting, meaning the LLM was not specifically fine-tuned on these scenarios. This suggests that the world models inherent in LLMs provide a significant advantage for generalization.

However, the study also identified a "toggling" problem: LLMs occasionally fluctuate rapidly between two behaviors (e.g., "overtake" vs. "wait"), leading to inconsistent vehicle movement. Furthermore, larger models did not always perform better; Llama-7B frequently outperformed Llama-13B, which occasionally made overly aggressive decisions resulting in collisions.

Conclusion

The interPlan benchmark exposes the limitations of current motion planners in handling the long tail of driving scenarios. While hybrid LLM-based systems show promise in providing the "common sense" necessary for complex navigation, significant work remains to stabilize their decision-making and synchronize their reasoning with real-time traffic dynamics. Future research will focus on fine-tuning LLMs with multimodal sensor data and auxiliary tasks to further improve safety and intent prediction.

Summary updated

Abstract

Real-world autonomous driving systems must make safe decisions in the face of rare and diverse traffic scenarios. Current state-of-the-art planners are mostly evaluated on real-world datasets like nuScenes (open-loop) or nuPlan (closed-loop). In particular nuPlan seems to be an expressive evaluation method since it is based on real-world data and closed-loop, yet it mostly covers basic driving scenarios. This makes it difficult to judge a planner’s capabilities to generalize to rarely-seen situations. Therefore, we propose a novel closed-loop benchmark interPlan containing several edge cases and challenging driving scenarios. We assess existing state-of-the-art planners on our benchmark and show that neither rule-based nor learning-based planners can safely navigate the interPlan scenarios.A recently evolving direction is the usage of foundation models like large language models (LLM) to handle generalization. We evaluate an LLM-only planner and introduce a novel hybrid planner that combines an LLM-based behavior planner with a rule-based motion planner that achieves state-of-the-art performance on our benchmark.
BibTeX
@inproceedings{hallgarten2024can,
  author = {Marcel Hallgarten and Julián Zapata and Martin Stoll and Katrin Renz and Andreas Zell},
  title = {{Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=SNsFbYgtfQ},
}
Figure