Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching

Meng Chu, Zhedong Zheng, Wei Ji, Tingyu Wang, Tat-Seng Chua

TL;DR

This paper introduces GeoText-1652, a novel benchmark dataset and a new "blending spatial matching" method for natural language-guided drone navigation. By providing detailed spatial annotations and a more robust matching approach, they aim to improve the precision and intuitiveness of drone control using language commands.

What Problem Does This Paper Address?

Navigating drones using natural language is a promising area with applications in disaster management, search and rescue, and remote sensing. However, a major hurdle is the lack of comprehensive, multi-modal datasets that precisely align visual and textual information. Existing datasets often focus on image-to-image matching across different viewpoints (like satellite and drone imagery) but don't adequately support fine-grained natural language instructions for drone control. This gap makes it difficult for models to understand and act upon detailed spatial descriptions, especially in complex, real-world scenarios where similar-looking objects abound. The paper aims to bridge this gap by creating a richer dataset and developing a more sophisticated matching mechanism.

Methods

The core of this work lies in two main contributions: a new dataset and a novel training objective.

GeoText-1652 Dataset: This dataset is built upon the existing University-1652 dataset. The key innovation is the addition of fine-grained, spatial-aware text annotations. This was achieved through an interactive human-computer process leveraging Large Language Models (LLMs). The process involves:

  • Modality Expansion: Using LLMs to generate image-level and region-level descriptions based on prompts. A "referee model" and human validation are used to ensure quality and filter out ambiguous or incorrect outputs.
  • Spatial Refinement: Employing text-based visual grounding models to identify bounding boxes corresponding to region-level descriptions. This phase also refines descriptions by adding vertical spatial terms (e.g., "upper left") and uses hand-designed rules to ensure accurate bounding box localization.

Blending Spatial Matching: This is a new optimization objective designed to improve region-level spatial relation matching. It goes beyond simply predicting bounding boxes for individual objects. Instead, it focuses on understanding the relative spatial relationships between multiple objects within an image. This is achieved by:

  • Grounding Prediction: A standard objective that predicts bounding boxes for identified objects.
  • Spatial Relation Matching: This novel component uses concatenated region features to predict the spatial relationship (e.g., "left," "right," "upper," "down") between pairs of objects, classifying into 9 categories. This helps the model understand context and disambiguate similar objects.

The overall framework combines image and text encoders with cross-attention modules, and the total loss function includes image-text contrastive loss (ITC), image-text matching loss (ITM), grounding loss, and the proposed spatial loss.

Datasets

  • GeoText-1652: The primary contribution, a new benchmark dataset for natural language-guided geolocalization. It extends the University-1652 dataset by adding detailed, spatial-aware text annotations, including image-level descriptions and region-level descriptions with corresponding bounding boxes. It comprises 1,652 buildings across three platforms (satellite, drone, ground cameras) from 72 universities.
  • University-1652: The base dataset used for GeoText-1652.

Key Insights and Ideas

  • The Importance of Fine-Grained Spatial Annotations: The paper strongly emphasizes that for natural language-guided drone navigation, simply identifying objects isn't enough. Understanding their precise spatial relationships (e.g., "the building to the left of the red-roofed house") is crucial for accurate localization and control.
  • LLMs as Powerful, but Imperfect, Annotation Tools: The authors leverage LLMs for efficient dataset creation but acknowledge their limitations (biases, hallucinations). Their proposed human-computer interaction framework, including a "referee model," highlights a practical approach to harness LLM capabilities while maintaining annotation quality.
  • Spatial Relation Matching is Key for Ambiguity Resolution: The "blending spatial matching" objective is shown to be critical for distinguishing between similar objects in complex scenes, where characteristics alone might be insufficient. The ability to understand "left," "right," "upper," and "down" relationships significantly enhances localization accuracy.
  • The "Large" Vision Model Challenge: The results suggest that even with large pre-trained models, fine-tuning on specialized datasets like GeoText-1652 is essential for achieving high performance in specific domains like aerial-view geolocalization.

Key Results

  • Dataset Effectiveness: Fine-tuning existing models (ALBEF, XVLM) on the GeoText-1652 dataset significantly improves their performance, demonstrating the dataset's value.
  • Method Superiority: The proposed "blending spatial matching" method achieves superior recall rates compared to baselines. For text-to-image retrieval, it shows improvements of +1.6% Recall@10 over XVLMfinetuned. For image-to-text retrieval, it achieves +1.8% Recall@10.
  • Ablation Studies:
    • Both grounding loss and spatial loss contribute positively to performance. The full model with both losses achieves the best results.
  • Using a combined training set ("Satellite + Drone + Ground") yields better performance than using individual platforms, indicating the benefit of diverse data.
  • The model is robust to small rotation perturbations, though performance degrades with larger rotations.

Limitations and Future Work

  • Rotation Perturbation: While the proposed method shows robustness to small rotations, performance degrades with larger rotation degrees. The authors note that this is expected as the text query might provide a "wrong" relative position.
  • Future Work: The paper implicitly suggests future work could focus on further improving robustness to various environmental conditions and exploring more complex natural language instructions for drone navigation. The GeoText-1652 dataset itself is a foundation for future research in this area.

Summary updated

Abstract

Navigating drones through natural language commands remains challenging due to the dearth of accessible multi-modal datasets and the stringent precision requirements for aligning visual and textual data. To address this pressing need, we introduce GeoText-1652, a new natural language-guided geolocalization benchmark. This dataset is systematically constructed through an interactive human-computer process leveraging Large Language Model (LLM) driven annotation techniques in conjunction with pre-trained vision models. GeoText-1652 extends the established University-1652 image dataset with spatial-aware text annotations, thereby establishing one-to-one correspondences between image, text, and bounding box elements. We further introduce a new optimization objective to leverage fine-grained spatial associations, called blending spatial matching, for region-level spatial relation matching. Extensive experiments reveal that our approach maintains a competitive recall rate comparing other prevailing cross-modality methods. This underscores the promising potential of our approach in elevating drone control and navigation through the seamless integration of natural language commands in real-world scenarios.
BibTeX
@inproceedings{chu2024towards,
  author = {Meng Chu and Zhedong Zheng and Wei Ji and Tingyu Wang and Tat-Seng Chua},
  title = {{Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73247-8\_13},
}
Figure