GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing

Zilun Zhang, Haozhan Shen, Tiancheng Zhao, Bin Chen, Zian Guan, Yuhao Wang, Xu Jia, Yuxiang Cai, Yongheng Shang, Jianwei YinarXiv 2025 · DOI · arXiv

Abstract

GeoRSMLLM is a multimodal large language model (MLLM) designed to unify diverse remote sensing (RS) tasks, ranging from basic scene classification to complex, multi-condition object detection and pixel-level segmentation. By introducing a "set-of-points" data representation and an intelligent condition parser, the model bridges the gap between high-level linguistic reasoning and precise spatial analysis required in geoscience applications.

Addressing the Remote Sensing Gap

While Vision-Language Models (VLMs) have advanced significantly in general computer vision, applying them to Remote Sensing presents unique challenges. Current RS models often struggle with:

  1. Complex Instructions: Queries involving multiple spatial or logical conditions (e.g., "find all planes on the east bank of the river").
  2. Pixel-Level Precision: Tasks like semantic segmentation or change detection require finer output than standard bounding boxes.
  3. Task Fragmentation: Most models are specialized for single tasks, lacking a unified framework for the full spectrum of RS cognitive requirements.

Methodology: Unifying Vision and Language

The authors propose GeoRSMLLM, built on the LLaVA architecture but heavily modified for the geoscience domain through several key innovations:

  • Unified Data Representation (Set-of-Points): Instead of task-specific formats, all spatial outputs—bounding boxes, segmentation masks, and change polygons—are represented as sets of coordinates. For instance, rotational boxes use four corner points, while complex shapes use polygons, allowing the LLM to treat spatial data as natural language sequences.
  • Condition Parser: To handle complex prompts, an independent LLM acts as a pre-processor, decomposing multi-condition instructions into a chain of simpler sub-tasks solved iteratively via chain-of-thought reasoning.
  • Model Backbone: The system utilizes Qwen2-7B as the language backbone and SigLIP as the vision tower, employing a dynamic high-resolution scheme to manage large-scale satellite imagery.
  • Self-Augmentation (Cyclic Referring): To strengthen the link between coordinates and descriptions, the authors use a "cyclic" strategy, converting detection/segmentation data into region captioning tasks and vice versa.

The RSVLTS Benchmark

The paper introduces the Remote Sensing Vision-Language Task Set (RSVLTS), which categorizes tasks into a hierarchy of increasing difficulty: Open-Vocabulary Tasks (OVT), Referring Expression Tasks (RET), and Described Object Tasks (DOT).

The RSVLTS task hierarchy categorizes remote sensing challenges into three levels of cognitive complexity: Open-Vocabulary, Referring Expression, and Described Object Tasks.

Key Insights and Cognitive Hierarchy

A central argument of the work is that RS tasks should be viewed through the lens of cognitive load. Moving from "what is in this image" (OVT) to "where is the object described by X" (RET) to "find the object that meets conditions A, B, and C" (DOT) requires fundamentally different reasoning capabilities. By reducing masks and boxes to "sets of points," the authors avoid task-specific decoders, enabling the LLM to handle spatial coordinates natively. Furthermore, for high-fidelity segmentation, the model integrates with SAM2 (Segment Anything Model 2), using generated points as prompts.

Performance and Versatility

GeoRSMLLM serves as a generalized solution, handling tasks that previously required separate specialized architectures, such as Referring Change Detection (RCD) and Geo-localization. The condition parser significantly improves instruction following, allowing the model to navigate complex spatial queries without the hallucinations typical of standard MLLMs.

Examples of GeoRSMLLM performing diverse tasks on a single remote sensing image, including conditional segmentation, zero-shot classification, and geo-localization.

Future Directions

While GeoRSMLLM provides a unified framework, the authors note that specialized models still hold an edge in specific areas like change detection. Future work will focus on refining the architecture to better leverage the point-based data and expanding the RSVLTS benchmark to include tasks like object counting and more granular geo-localization.

Summary updated

Abstract

The application of Vision-Language Models (VLMs) in remote sensing (RS) has demonstrated significant potential in traditional tasks such as scene classification, object detection, and image captioning. However, current models, which excel in Referring Expression Comprehension (REC), struggle with tasks involving complex instructions (e.g., exists multiple conditions) or pixel-level operations like segmentation and change detection. In this white paper, we provide a comprehensive hierarchical summary of vision-language tasks in RS, categorized by the varying levels of cognitive capability required. We introduce the Remote Sensing Vision-Language Task Set (RSVLTS), which includes Open-Vocabulary Tasks (OVT), Referring Expression Tasks (RET), and Described Object Tasks (DOT) with increased difficulty, and Visual Question Answering (VQA) aloneside. Moreover, we propose a novel unified data representation using a set-of-points approach for RSVLTS, along with a condition parser and a self-augmentation strategy based on cyclic referring. These features are integrated into the GeoRSMLLM model, and this enhanced model is designed to handle a broad range of tasks of RSVLTS, paving the way for a more generalized solution for vision-language tasks in geoscience and remote sensing.
BibTeX
@misc{zhang2025georsmllm,
  author = {Zilun Zhang and Haozhan Shen and Tiancheng Zhao and Bin Chen and Zian Guan and Yuhao Wang and Xu Jia and Yuxiang Cai and Yongheng Shang and Jianwei Yin},
  title = {{GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing}},
  year = {2025},
  eprint = {2503.12490},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.12490},
}
Figure