Language-Guided Hybrid Representation Learning for Visual Grounding on Remote Sensing Images

Biao Liu, Xu Liu, Lingling Li, Licheng Jiao, Fang Liu, Xinyu Sun, Youlin Huang

Presented at IJCAI-25, LGFormer is a novel Transformer-based framework designed to solve the "where is it?" problem in satellite and aerial imagery. By utilizing "hybrid queries" that combine linguistic guidance with multi-scale visual features, the authors achieve state-of-the-art performance in locating specific objects within complex remote sensing scenes.

The Challenge of Remote Sensing Visual Grounding

Visual Grounding (VG) involves locating an object in an image based on a natural language description. While VG is well-established for natural images, Remote Sensing Visual Grounding (RSVG) is significantly harder. Satellite images cover massive areas, and objects of interest—like airplanes or storage tanks—can vary wildly in scale. Furthermore, these images are cluttered with complex backgrounds. Existing models often suffer from "attention drift," where the model's internal search mechanism fails to correlate text descriptions with the correct visual features amidst background noise.

LGFormer Architecture

The authors propose the Language-Guided hybrid representation learning Transformer (LGFormer) to address these challenges. Its architecture consists of four main components designed to align modalities and focus search.

The overall architecture of LGFormer, illustrating the dual-encoder backbone, multi-scale cross-modal enhancement, and the generation of hybrid queries.
  1. Dual-Encoder Backbone: The model uses a parallel visual encoder (CNN + Vision Transformer) to capture both local textures and global context, alongside a BERT-based text encoder.
  2. Multi-Scale Cross-Modal Enhancement (MSCM): This module uses cross-attention to align text and visual features in a shared semantic space, ensuring linguistic terms correspond to visual patterns.
  3. Language-Guided Visual Feature Filtering: Instead of processing every pixel, the model uses text features to "filter" the visual field, selecting the top k visual embeddings most likely to contain the target.
  4. Hybrid Queries & Adaptive Fusion: LGFormer initializes its search using a combination of sentence-level text embeddings and filtered visual features. These "Hybrid Queries" pass through an Adaptive Multimodal Feature Fusion module using multi-scale deformable attention to process the image efficiently at different resolutions.

Key Insights: Initialization and Filtering

Traditional Transformers use "learnable queries" that start as random noise. By using Hybrid Queries, LGFormer gives the decoder a strong prior of what it is looking for and where it might be, significantly reducing search errors. Additionally, using language to prune the visual search space early on prevents the model from being overwhelmed by background interference in vast satellite images.

Performance and Results

The model was evaluated on two primary benchmarks: DIOR-RSVG (17,402 images) and OPT-RSVG, a more challenging dataset with a wider range of target scales. LGFormer achieved the best results to date on both datasets, outperforming previous state-of-the-art models like LQVG and LPVA.

Qualitative result showing LGFormer's ability to accurately localize a specific vehicle within a cluttered environment.

Qualitative analysis shows that LGFormer can distinguish between identical-looking objects by better understanding spatial qualifiers in the text (e.g., "the vehicle in the middle").

Limitations and Future Outlook

Despite its success, LGFormer is currently designed to locate only one object at a time. Future work will explore multi-object localization and improving robustness to vague descriptions that lack precise spatial qualifiers. The team also plans to explore collaborative learning strategies for multi-source images, combining different types of satellite data.

Reference: Liu, B., Liu, X., et al. (2025). Language-Guided Hybrid Representation Learning for Visual Grounding on Remote Sensing Images. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI-25).

Summary updated

Abstract

Visual grounding (VG) refers to detecting the specific objects in images based on linguistic expressions, and it has profound significance in the advanced interpretation of natural images. In remote sensing image interpretation, visual grounding is limited by characteristics such as the complex scenes and diverse object sizes. To solve this problem, we propose a novel remote sensing visual grounding (RSVG) framework, named language-guided hybrid representation learning Transformer (LGFormer). Specifically, we designed a multimodal dual-encoder Transformer structure called the adaptive multimodal feature fusion module. This structure innovatively integrates text and visual features as hybrid queries, enabling early-stage decoding queries to perceive the target position accurately. Then, the different modal information from the dual encoders is aggregated by hybrid queries to obtain the final object embedding for coordinate regression. Besides, a multi-scale cross-modal feature enhancement module (MSCM) is designed to enhance the self-representation of the extracted text and visual features and align them semantically. As for the hybrid queries, we use linguistic guidance to select visual features as the visual part and sentence-level features as the textual part. Finally, the LGFormer model we designed achieved the best results compared to existing models on the DIOR-RSVG and OPT-RSVG datasets.
BibTeX
@inproceedings{liu2025language,
  author = {Biao Liu and Xu Liu and Lingling Li and Licheng Jiao and Fang Liu and Xinyu Sun and Youlin Huang},
  title = {{Language-Guided Hybrid Representation Learning for Visual Grounding on Remote Sensing Images}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/174},
}
Figure