Look Only Where You Plan for Real-Time End-to-End Urban Driving

Haohua Que, Tianle Zhu, Qianyi Wu, Zhipeng Bao, Handong YaoarXiv 2026 · arXiv

TL;DR

LOWP (Look Only Where You Plan) is a new end-to-end driving planner that ditches the computationally expensive dense Bird’s-Eye-View (BEV) grids used by most modern systems. By using just 32 learnable queries to "read" camera features through known calibration, it achieves state-of-the-art planning accuracy with 47% fewer parameters and a 1.7x speedup over its dense counterparts.

What Problem Does This Paper Address?

Most high-performing end-to-end autonomous driving planners follow a "lift-splat-shoot" philosophy: they take multi-camera images, lift them into a dense 3D or BEV grid, and then "shoot" a trajectory through that grid. While effective, this dense reconstruction is a massive resource hog—often accounting for over 60% of a model's parameters.

The authors of LOWP ask a provocative question: Do we actually need to reconstruct the entire scene to plan a safe path? In urban driving, much of the environment (like the sky or distant buildings) is irrelevant to the immediate 3-second trajectory. LOWP aims to prove that a sparse, task-specific representation is not only more efficient but potentially more accurate for planning.

Methods

LOWP replaces the dense BEV pathway with a sparse architecture featuring two specialized "experts" (readers), each handling 16 queries:

  1. The Deformable Reader: This expert uses metric 3D anchors projected into the camera views. It samples image features at specific points around these anchors. This provides a strong geometric prior, making it particularly effective at handling complex turns.
  2. The Global Reader: This expert uses geometry-aware attention. Instead of fixed anchors, it attends to image cells labeled with their viewing rays. This allows it to capture a broader context, which is useful for straight-line driving.
The LOWP architecture uses two experts to process camera features without building a dense BEV grid.

The outputs of these two readers are mixed based on the high-level driving command (e.g., "turn left"). Beyond the architecture, the authors also introduce a "no host sync" implementation strategy, removing bottlenecks between the GPU and CPU to ensure the model runs at true real-time speeds.

Datasets

The model was evaluated across three distinct regimes to ensure robustness:

  • nuScenes: Used for open-loop evaluation (predicting trajectories against human ground truth).
  • NAVSIM: A pseudo-simulation benchmark that scores trajectories based on safety and progress.
  • Bench2Drive: A closed-loop simulator where the planner must actually control the vehicle in a reactive environment.

Key Insights and Ideas

  • Sparsity is Sufficient: The most striking finding is that just 32 tokens (queries) can represent enough environmental context to drive as well as—or better than—a model using millions of BEV grid cells.
  • Geometric Specialization: The authors found that "where" you look matters differently depending on the maneuver. The anchor-based deformable reader significantly outperformed the global reader on turns (7 collisions vs. 24 in a test set), proving that explicit 3D grounding is vital for non-linear paths.
  • The "BEV Tax": By avoiding dense BEV construction, the model avoids the need for expensive depth supervision or complex object/map labels during training, simplifying the pipeline significantly.
  • Latency is more than FLOPs: The paper highlights that real-world latency is often driven by "kernel launches" and host-device synchronization rather than just raw computation. Their optimizations for batch-one inference are a practical blueprint for deployment.

Key Results

LOWP demonstrates that "less is more" in the context of planning:

  • Efficiency: It uses 42.9M parameters compared to the 80.8M of its dense counterpart, ResWorld.
  • Speed: On the same GPU, it runs 1.7x faster (24.4ms vs 41.3ms latency).
  • Accuracy: On nuScenes, it improved the average L2 displacement error from 0.302m to 0.296m.
  • Closed-Loop Performance: On Bench2Drive, it achieved a driving score of 65.51, outperforming several much larger models.
A comparison of accuracy vs. speed across different benchmarks shows LOWP (red squares) pushing the Pareto front.

Qualitatively, the model shows a better ability to "hug" turns compared to dense planners, which sometimes under-steer or cut corners.

Qualitative comparison showing LOWP successfully navigating turns where the dense ResWorld model fails or collides.

Limitations and Future Work

While LOWP is highly efficient, the authors note a few areas for improvement:

  • Standstill Deadlock: In closed-loop testing, the model occasionally suffered from "standstill failures" where the brake-prediction head would hold the car still even when the trajectory planner wanted to move.
  • Supervision Gaps: Because the model doesn't use object or map labels, it might lack the semantic "understanding" required for extremely complex social interactions (e.g., yielding to a specific pedestrian's intent).
  • Hardware Comparisons: The paper notes that comparing "FPS" across different research papers is notoriously difficult due to varying hardware and reporting standards, suggesting a need for more standardized benchmarking in the field.

Summary updated

Abstract

Dense bird's-eye-view (BEV) construction accounts for 60% of the parameters in a strong recent driving planner. This paper presents LOWP, a planner that looks only where it plans: two sets of sixteen learnable queries read the cameras through the known calibration, one by sampling image features at projected 3D anchors and one by attending to features labeled with their viewing rays, so no dense BEV is built and no object or map is supervised. Evaluations on open-loop nuScenes, pseudo-simulated NAVSIM and closed-loop Bench2Drive show competitive planning with a compact representation. On nuScenes, average L2 changes from 0.302 to 0.296 m with 53% of the dense counterpart's parameters and 1.7 times its speed on the same GPU; the anchor-based reader performs better on turns than the global reader. On NAVSIM, scoring the planner's own trajectories alongside a vocabulary gives 85.9 PDMS without retraining. The closed-loop control configuration scores 65.5 on 218 routes, with a brake-induced standstill failure whose release rule has an inconclusive aggregate benefit. We distinguish measured latency from hardware-transferred estimates, which do not establish a common-hardware ranking.
BibTeX
@misc{que2026look,
  author = {Haohua Que and Tianle Zhu and Qianyi Wu and Zhipeng Bao and Handong Yao},
  title = {{Look Only Where You Plan for Real-Time End-to-End Urban Driving}},
  year = {2026},
  eprint = {2611.10264},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10264},
}
Figure