Abstract
This paper introduces GeoText-1652, a comprehensive benchmark designed to facilitate drone geolocalization through natural language instructions. Unlike traditional image-to-image matching, GeoText-1652 leverages over 300,000 textual descriptions to bridge the gap between human-readable spatial commands and fine-grained visual scenes. We propose a "blending spatial matching" technique that incorporates grounding and spatial losses to enhance the model's understanding of relative object positions. Experimental results demonstrate significant improvements in cross-modal retrieval, achieving a 31.2% Recall@10 and highlighting the importance of spatial reasoning in autonomous navigation.
Introduction
Drone geolocalization—the process of a drone determining its coordinates by analyzing ground imagery—traditionally relies on matching a real-time camera feed against a database of satellite photos. While effective, this "image-to-image" paradigm is limited in dynamic or emergency scenarios, such as search and rescue, where a human operator might only be able to provide verbal descriptions (e.g., "Find the red-roofed building with a parking lot on its left").
Transitioning to language-guided navigation faces two primary challenges: the scarcity of large-scale datasets pairing drone imagery with spatially-aware text, and the fine-grained ambiguity of urban environments where many structures look identical. To resolve these, models must understand not just object identity, but also relative spatial relationships.
Methodology
We propose a cross-modal retrieval framework that treats drone navigation as a matching problem between text and images. The core innovation is the Blending Spatial Matching objective, which moves beyond global image features to a more granular understanding of the scene.
Spatial Matching and Grounding
The model is trained using two specialized loss functions:
- Grounding Loss: The model learns to predict bounding boxes for specific objects mentioned in the text queries.
- Spatial Loss: This "secret sauce" requires the model to classify the relative positions of objects using a 9-class spatial relationship grid (e.g., Top-Left, Middle-Right). By forcing the model to distinguish these relationships, it develops a deeper understanding of scene layout.
Semi-Automatic Annotation Pipeline
To scale the dataset efficiently, we utilized a Human-Computer Interaction (HCI) pipeline. Visual-LLMs (such as GPT-4V) generated initial descriptions, which were then filtered by a "Referee Model" to eliminate hallucinations. Finally, human annotators performed quality checks to ensure spatial accuracy.
The GeoText-1652 Dataset
GeoText-1652 extends the University-1652 dataset to provide a massive scale for multi-modal geolocalization. It includes 276,045 text-bounding box pairs and 316,335 total descriptions across drone, satellite, and ground-level views.
Unlike previous datasets that provide generic labels, GeoText-1652 offers "Dense Descriptions." These descriptions focus on the surroundings of a target, recognizing that context (e.g., "the building next to the circular pond") is often the only way to distinguish between visually similar targets in a dense campus or city environment.
Key Results and Insights
The proposed model was evaluated on tasks including Drone Navigation (Text-to-Image) and Target Localization (Image-to-Text).
- Performance: The model achieved a 31.2% Recall@10 for text queries, outperforming established multi-modal models like ALBEF and X-VLM.
- Spatial Awareness: Ablation studies showed that the spatial matching component provided a +1.6% boost in Recall@10 over the baseline, proving that spatial reasoning is a distinct and necessary skill for drone AI.
- Zero-Shot Capabilities: Qualitatively, the model successfully identified objects in "unseen" environments, such as finding boats in a harbor despite being trained primarily on university campus data.
Limitations and Future Work
A significant remaining challenge is the "Rotation Problem." Current spatial terms like "left" and "right" are fixed relative to the image frame. If a drone rotates, these terms lose their absolute meaning relative to a map. Future iterations will need to incorporate the drone's heading (compass direction) to maintain spatial consistency. Additionally, the research aims to move from retrieval-based localization to active Vision-Language Navigation (VLN), where drones can execute flight paths in real-time based on verbal commands.
Summary updated