Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatially Relation Matching

Meng Chu, Zhedong Zheng, Wei Ji, Tat-Seng Chua

Abstract

This paper introduces GeoText-1652, a comprehensive benchmark designed to facilitate drone geolocalization through natural language instructions. Unlike traditional image-to-image matching, GeoText-1652 leverages over 300,000 textual descriptions to bridge the gap between human-readable spatial commands and fine-grained visual scenes. We propose a "blending spatial matching" technique that incorporates grounding and spatial losses to enhance the model's understanding of relative object positions. Experimental results demonstrate significant improvements in cross-modal retrieval, achieving a 31.2% Recall@10 and highlighting the importance of spatial reasoning in autonomous navigation.

Introduction

Drone geolocalization—the process of a drone determining its coordinates by analyzing ground imagery—traditionally relies on matching a real-time camera feed against a database of satellite photos. While effective, this "image-to-image" paradigm is limited in dynamic or emergency scenarios, such as search and rescue, where a human operator might only be able to provide verbal descriptions (e.g., "Find the red-roofed building with a parking lot on its left").

Transitioning to language-guided navigation faces two primary challenges: the scarcity of large-scale datasets pairing drone imagery with spatially-aware text, and the fine-grained ambiguity of urban environments where many structures look identical. To resolve these, models must understand not just object identity, but also relative spatial relationships.

Figure 1: The proposed pipeline for natural language-guided geolocalization, moving from text queries to image and geolocation retrieval.

Methodology

We propose a cross-modal retrieval framework that treats drone navigation as a matching problem between text and images. The core innovation is the Blending Spatial Matching objective, which moves beyond global image features to a more granular understanding of the scene.

Spatial Matching and Grounding

The model is trained using two specialized loss functions:

  1. Grounding Loss: The model learns to predict bounding boxes for specific objects mentioned in the text queries.
  2. Spatial Loss: This "secret sauce" requires the model to classify the relative positions of objects using a 9-class spatial relationship grid (e.g., Top-Left, Middle-Right). By forcing the model to distinguish these relationships, it develops a deeper understanding of scene layout.

Semi-Automatic Annotation Pipeline

To scale the dataset efficiently, we utilized a Human-Computer Interaction (HCI) pipeline. Visual-LLMs (such as GPT-4V) generated initial descriptions, which were then filtered by a "Referee Model" to eliminate hallucinations. Finally, human annotators performed quality checks to ensure spatial accuracy.

The GeoText-1652 Dataset

GeoText-1652 extends the University-1652 dataset to provide a massive scale for multi-modal geolocalization. It includes 276,045 text-bounding box pairs and 316,335 total descriptions across drone, satellite, and ground-level views.

Figure 3: Comparison of caption complexity levels. GeoText-1652 focuses on "Dense Description," providing region-specific spatial details that go beyond simple category labels.

Unlike previous datasets that provide generic labels, GeoText-1652 offers "Dense Descriptions." These descriptions focus on the surroundings of a target, recognizing that context (e.g., "the building next to the circular pond") is often the only way to distinguish between visually similar targets in a dense campus or city environment.

Key Results and Insights

The proposed model was evaluated on tasks including Drone Navigation (Text-to-Image) and Target Localization (Image-to-Text).

  • Performance: The model achieved a 31.2% Recall@10 for text queries, outperforming established multi-modal models like ALBEF and X-VLM.
  • Spatial Awareness: Ablation studies showed that the spatial matching component provided a +1.6% boost in Recall@10 over the baseline, proving that spatial reasoning is a distinct and necessary skill for drone AI.
  • Zero-Shot Capabilities: Qualitatively, the model successfully identified objects in "unseen" environments, such as finding boats in a harbor despite being trained primarily on university campus data.

Limitations and Future Work

A significant remaining challenge is the "Rotation Problem." Current spatial terms like "left" and "right" are fixed relative to the image frame. If a drone rotates, these terms lose their absolute meaning relative to a map. Future iterations will need to incorporate the drone's heading (compass direction) to maintain spatial consistency. Additionally, the research aims to move from retrieval-based localization to active Vision-Language Navigation (VLN), where drones can execute flight paths in real-time based on verbal commands.

Summary updated

Abstract

Navigating drones through natural language commands remains challenging due to the dearth of accessible multi-modal datasets and the stringent precision requirements for aligning visual and textual data. To address this pressing need, we introduce GeoText-1652, a new natural language-guided geo-localization benchmark. This dataset is systematically constructed through an interactive human-computer process leveraging Large Language Model (LLM) driven annotation techniques in conjunction with pre-trained vision models. GeoText-1652 extends the established University-1652 image dataset with spatial-aware text annotations, thereby establishing one-to-one correspondences between image, text, and bounding box elements. We further introduce a new optimization objective to leverage fine-grained spatial associations, called blending spatial matching, for region-level spatial relation matching. Extensive experiments reveal that our approach maintains a competitive recall rate comparing other prevailing cross-modality methods. This underscores the promising potential of our approach in elevating drone control and navigation through the seamless integration of natural language commands in real-world scenarios.
BibTeX
@inproceedings{chu2024towards,
  author = {Meng Chu and Zhedong Zheng and Wei Ji and Tat-Seng Chua},
  title = {{Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatially Relation Matching}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73247-8\_13},
}
Figure