TL;DR
LOWP (Look Only Where You Plan) is a new end-to-end driving planner that ditches the computationally expensive dense Bird’s-Eye-View (BEV) grids used by most modern systems. By using just 32 learnable queries to "read" camera features through known calibration, it achieves state-of-the-art planning accuracy with 47% fewer parameters and a 1.7x speedup over its dense counterparts.
What Problem Does This Paper Address?
Most high-performing end-to-end autonomous driving planners follow a "lift-splat-shoot" philosophy: they take multi-camera images, lift them into a dense 3D or BEV grid, and then "shoot" a trajectory through that grid. While effective, this dense reconstruction is a massive resource hog—often accounting for over 60% of a model's parameters.
The authors of LOWP ask a provocative question: Do we actually need to reconstruct the entire scene to plan a safe path? In urban driving, much of the environment (like the sky or distant buildings) is irrelevant to the immediate 3-second trajectory. LOWP aims to prove that a sparse, task-specific representation is not only more efficient but potentially more accurate for planning.
Methods
LOWP replaces the dense BEV pathway with a sparse architecture featuring two specialized "experts" (readers), each handling 16 queries:
- The Deformable Reader: This expert uses metric 3D anchors projected into the camera views. It samples image features at specific points around these anchors. This provides a strong geometric prior, making it particularly effective at handling complex turns.
- The Global Reader: This expert uses geometry-aware attention. Instead of fixed anchors, it attends to image cells labeled with their viewing rays. This allows it to capture a broader context, which is useful for straight-line driving.
The outputs of these two readers are mixed based on the high-level driving command (e.g., "turn left"). Beyond the architecture, the authors also introduce a "no host sync" implementation strategy, removing bottlenecks between the GPU and CPU to ensure the model runs at true real-time speeds.
Datasets
The model was evaluated across three distinct regimes to ensure robustness:
- nuScenes: Used for open-loop evaluation (predicting trajectories against human ground truth).
- NAVSIM: A pseudo-simulation benchmark that scores trajectories based on safety and progress.
- Bench2Drive: A closed-loop simulator where the planner must actually control the vehicle in a reactive environment.
Key Insights and Ideas
- Sparsity is Sufficient: The most striking finding is that just 32 tokens (queries) can represent enough environmental context to drive as well as—or better than—a model using millions of BEV grid cells.
- Geometric Specialization: The authors found that "where" you look matters differently depending on the maneuver. The anchor-based deformable reader significantly outperformed the global reader on turns (7 collisions vs. 24 in a test set), proving that explicit 3D grounding is vital for non-linear paths.
- The "BEV Tax": By avoiding dense BEV construction, the model avoids the need for expensive depth supervision or complex object/map labels during training, simplifying the pipeline significantly.
- Latency is more than FLOPs: The paper highlights that real-world latency is often driven by "kernel launches" and host-device synchronization rather than just raw computation. Their optimizations for batch-one inference are a practical blueprint for deployment.
Key Results
LOWP demonstrates that "less is more" in the context of planning:
- Efficiency: It uses 42.9M parameters compared to the 80.8M of its dense counterpart, ResWorld.
- Speed: On the same GPU, it runs 1.7x faster (24.4ms vs 41.3ms latency).
- Accuracy: On nuScenes, it improved the average L2 displacement error from 0.302m to 0.296m.
- Closed-Loop Performance: On Bench2Drive, it achieved a driving score of 65.51, outperforming several much larger models.
Qualitatively, the model shows a better ability to "hug" turns compared to dense planners, which sometimes under-steer or cut corners.
Limitations and Future Work
While LOWP is highly efficient, the authors note a few areas for improvement:
- Standstill Deadlock: In closed-loop testing, the model occasionally suffered from "standstill failures" where the brake-prediction head would hold the car still even when the trajectory planner wanted to move.
- Supervision Gaps: Because the model doesn't use object or map labels, it might lack the semantic "understanding" required for extremely complex social interactions (e.g., yielding to a specific pedestrian's intent).
- Hardware Comparisons: The paper notes that comparing "FPS" across different research papers is notoriously difficult due to varying hardware and reporting standards, suggesting a need for more standardized benchmarking in the field.
Summary updated