TL;DR
This paper identifies a critical "reaching-is-not-finding" gap in aerial vision-language navigation, where drones successfully navigate to a landmark but fail to identify the specific target object. The authors introduce MAP-VP, a plug-and-play module that draws map-anchored landmark outlines directly onto the drone's view as visual prompts, roughly doubling the success rates of state-of-the-art navigators by teaching a fine-tuned VLM to use these landmarks as spatial references.
What Problem Does This Paper Address?
Aerial Vision-Language Navigation (VLN) requires a drone to locate a specific object (e.g., "a dark red vehicle") based on its spatial relationship to landmarks (e.g., "parked along Grafton Street across from a dark blue car").
The authors discovered a systemic failure they call the "reaching-is-not-finding" gap. Current navigation models are surprisingly good at reaching the correct neighborhood by following landmark names, but they are terrible at the "last mile." Once the drone arrives, it often stops near the target but fails to pick the correct instance among similar-looking objects. A simple target-swap test proved that most navigators essentially ignore the target description at the end, relying almost entirely on landmark proximity.
Methods
The authors propose MAP-VP (Map-Anchored Visual Prompting), a modular addition that runs after a primary navigator has finished its flight.
- Visual Prompting: MAP-VP takes the drone's final top-down view and overlays the outlines and names of landmarks mentioned in the instruction. These polygons are projected from existing map data (like OpenStreetMap).
- Fine-tuned VLM: A 4B-parameter Vision-Language Model (Qwen3-VL) is fine-tuned using LoRA to interpret these visual prompts. Crucially, the model learns to treat the yellow outlines as references rather than the answer itself.
- Automatic Supervision: Training data is generated automatically from benchmark annotations. The model is trained to output a bounding box for the target or a "null" response if the target is not visible.
- Look-Again (LA) Mechanism: If the model returns "null," the drone climbs from 50m to 100m to get a wider field of view and tries again.
- Execution: Once a target is identified, the drone performs a final short flight to the center of the predicted box.
Datasets
The study primarily uses CityNav, a large-scale benchmark for aerial VLN featuring real-world city scans of Birmingham and Cambridge. The authors utilized the revised annotations from HETT, which corrected landmark lists and filtered out noisy episodes. Evaluation was conducted on the test-unseen split to ensure the model could generalize to entirely new urban environments.
Key Insights and Ideas
- Navigators are Landmark-Seekers: A simple "landmark-centroid" rule (flying to the average position of named landmarks) outperformed every trained navigator tested, achieving a 40.2% success rate. This highlights that existing models prioritize landmark proximity over target identification.
- The Synergy of Anchors and Supervision: Drawing landmarks on the image (anchors) and fine-tuning the model (supervision) are both necessary. Without fine-tuning, zero-shot VLMs tend to mistake the marked landmarks for the target. Without the visual anchors, the model lacks the geometric context to resolve spatial relations like "across from" or "in front of."
- Spatial Grounding via Attention: Attention analysis showed that when landmarks are drawn, the model's internal attention effectively grounds the landmark names to the specific pixels within the outlines, which then allows it to shift focus to the correct target nearby.
Key Results
MAP-VP provides a massive performance boost across six different navigator architectures without requiring them to be retrained:
- HETT (State-of-the-art): Success Rate (SR) jumped from 25.9% to 54.6%.
- FlightGPT (VLM-based): SR increased from 24.2% to 48.9%.
- Recovery Rate: For episodes where the target was already in the drone's final view but the navigator failed, MAP-VP recovered 50–62% of those failures.
- Precision: At a stricter 10m success threshold (SR@10), HETT's performance nearly quadrupled, rising from 10.1% to 38.0%.
Limitations and Future Work
- Range Limit: MAP-VP is a "last mile" solution. If the base navigator stops more than 75m away from the target, the target is usually not in view, and MAP-VP cannot help.
- Counting Weakness: The model still struggles with instructions involving counting (e.g., "the 8th house to the left"). Success rates on counting-based instructions were about 10 points lower than average.
- Map Dependency: The system assumes access to accurate GPS and landmark polygons. Real-world errors in pose or map data could shift the visual prompts and mislead the model.
- Future Directions: Integrating this visual prompting logic directly into the navigation phase (rather than just the final step) could potentially help drones that currently fail to even reach the target's vicinity.
Summary updated