XeMap: Contextual Referring in Large-Scale Remote Sensing Environments

Yuxi Li, Lu Si, Yujie Hou, Chengaung Liu, Bin Li, Hongjian Fang, Jun ZhangarXiv 2025 · DOI · arXiv

Introduction: The Semantic Gap in Remote Sensing

Remote Sensing (RS) imagery, derived from high-resolution satellite and drone platforms, has traditionally been analyzed through two primary lenses: image-level classification and object-level detection. Image-level analysis provides a broad overview (e.g., "this is a residential area") but lacks spatial precision. Conversely, object-level detection identifies individual entities (e.g., "this is a car") but often fails to capture the broader contextual relationships between landmarks.

This dichotomy leaves a significant "semantic gap" at the mid-scale. Practical applications often require localizing complex entities defined by their relationships to surroundings—such as "the building complex adjacent to the playground" or "the parking lot between the industrial zone and the highway." To address this, researchers have introduced the conteXtual referring Map (XeMap) task, focusing on precise localization of regions described by complex, context-dependent natural language.

Comparison of XeMap with traditional image-level and object-level tasks in remote sensing.

The XeMap-Network Architecture

To tackle the challenges of multi-hop reasoning and multi-scale localization, the authors proposed the XeMap-Network. This end-to-end architecture is designed to process vast geographic scenes and interpret intricate textual queries through three specialized layers.

1. Encoder Layer

The network utilizes a dual-stream encoding process. Visual features are extracted using a Swin Transformer, which is adept at capturing hierarchical spatial information. Textual instructions are processed via a BERT encoder, transforming natural language queries into high-dimensional textual tokens.

2. Fusion Layer

Rather than simple feature concatenation, the Fusion Layer employs sophisticated cross-modal attention mechanisms. This allows the model to perform bidirectional reasoning: the text "looks" at relevant image regions, while the image features are weighted based on textual cues. To maintain efficiency across high-resolution inputs, the layer utilizes deformable self-attention, which focuses on a sparse set of sampling points rather than the entire feature map.

3. HMSA Module

The Hierarchical Multi-Scale Semantic Alignment (HMSA) module is the core innovation of the network. It aligns visual features across multiple scales—from fine-grained details to broad contextual textures—with a unified text semantic vector. This ensures that the model can accurately localize entities regardless of their physical size, producing a "correlation map" or heatmap that indicates where the textual description applies.

Detailed architecture of the XeMap-Network, highlighting the cross-modal fusion and hierarchical alignment modules.

Dataset and Training Methodology

A significant hurdle in contextual referring is the lack of specialized training data. The authors addressed this by creating XeMap-Set, a "mega-dataset" synthesized from prominent RS sources including VisDrone, DIOR-R, and DOTA v2.0.

  • Scale and Complexity: The training set comprises over 86,000 images and nearly 800,000 queries.
  • Reasoning Challenges: The test set was manually curated to include 1,129 complex queries involving "multi-hop" spatial reasoning (e.g., "A is next to B which is near C") and "multi-ref" expressions (referring to multiple distinct regions simultaneously).

To facilitate training on this data, the Multiscale Correlation Map Generation (MCMG) method was developed. This technique automatically converts standard bounding box or polygon annotations into the soft heatmaps required for semantic-level referring, allowing the model to learn from existing object-level datasets.

Experimental Results

The XeMap-Network was benchmarked against state-of-the-art (SOTA) methods, including CLIP-SeLo, in a zero-shot setting. The results demonstrate a clear superiority in both accuracy and operational efficiency:

  • Accuracy: The network achieved a Unified Metric ($R_{mi}$) score of 0.5789, a significant improvement over the previous SOTA of 0.4903.
  • Precision: It recorded a Significant Area Proportion ($R_{su}$) of 0.8651, indicating that the generated heatmaps are highly concentrated on the correct ground-truth regions.
  • Efficiency: Processing the entire test set of 1,129 queries took only 201 seconds on a single RTX 3090 GPU, representing a 10x speedup over competing CLIP-based architectures.

Conclusion and Future Directions

XeMap-Network successfully bridges the gap between broad image classification and narrow object detection in remote sensing. By leveraging hierarchical alignment and cross-modal fusion, it enables sophisticated spatial reasoning over large-scale imagery.

Future research may focus on expanding the framework to multi-spectral data (e.g., infrared-visible fusion) to support all-weather monitoring. Additionally, as general-purpose vision and language backbones evolve, integrating more powerful encoders could further enhance the model's ability to parse extremely complex spatial chains and nuanced geographic descriptions.

Summary updated

Abstract

Advancements in remote sensing (RS) imagery have provided high-resolution detail and vast coverage, yet existing methods, such as image-level captioning/retrieval and object-level detection/segmentation, often fail to capture mid-scale semantic entities essential for interpreting large-scale scenes. To address this, we propose the conteXtual referring Map (XeMap) task, which focuses on contextual, fine-grained localization of text-referred regions in large-scale RS scenes. Unlike traditional approaches, XeMap enables precise mapping of mid-scale semantic entities that are often overlooked in image-level or object-level methods. To achieve this, we introduce XeMap-Network, a novel architecture designed to handle the complexities of pixel-level cross-modal contextual referring mapping in RS. The network includes a fusion layer that applies self- and cross-attention mechanisms to enhance the interaction between text and image embeddings. Furthermore, we propose a Hierarchical Multi-Scale Semantic Alignment (HMSA) module that aligns multiscale visual features with the text semantic vector, enabling precise multimodal matching across large-scale RS imagery. To support XeMap task, we provide a novel, annotated dataset, XeMap-set, specifically tailored for this task, overcoming the lack of XeMap datasets in RS imagery. XeMap-Network is evaluated in a zero-shot setting against state-of-the-art methods, demonstrating superior performance. This highlights its effectiveness in accurately mapping referring regions and providing valuable insights for interpreting large-scale RS environments.
BibTeX
@misc{li2025xemap,
  author = {Yuxi Li and Lu Si and Yujie Hou and Chengaung Liu and Bin Li and Hongjian Fang and Jun Zhang},
  title = {{XeMap: Contextual Referring in Large-Scale Remote Sensing Environments}},
  year = {2025},
  eprint = {2505.00738},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.00738},
}
Figure