Abstract
GeoRSMLLM is a multimodal large language model (MLLM) designed to unify diverse remote sensing (RS) tasks, ranging from basic scene classification to complex, multi-condition object detection and pixel-level segmentation. By introducing a "set-of-points" data representation and an intelligent condition parser, the model bridges the gap between high-level linguistic reasoning and precise spatial analysis required in geoscience applications.
Addressing the Remote Sensing Gap
While Vision-Language Models (VLMs) have advanced significantly in general computer vision, applying them to Remote Sensing presents unique challenges. Current RS models often struggle with:
- Complex Instructions: Queries involving multiple spatial or logical conditions (e.g., "find all planes on the east bank of the river").
- Pixel-Level Precision: Tasks like semantic segmentation or change detection require finer output than standard bounding boxes.
- Task Fragmentation: Most models are specialized for single tasks, lacking a unified framework for the full spectrum of RS cognitive requirements.
Methodology: Unifying Vision and Language
The authors propose GeoRSMLLM, built on the LLaVA architecture but heavily modified for the geoscience domain through several key innovations:
- Unified Data Representation (Set-of-Points): Instead of task-specific formats, all spatial outputs—bounding boxes, segmentation masks, and change polygons—are represented as sets of coordinates. For instance, rotational boxes use four corner points, while complex shapes use polygons, allowing the LLM to treat spatial data as natural language sequences.
- Condition Parser: To handle complex prompts, an independent LLM acts as a pre-processor, decomposing multi-condition instructions into a chain of simpler sub-tasks solved iteratively via chain-of-thought reasoning.
- Model Backbone: The system utilizes Qwen2-7B as the language backbone and SigLIP as the vision tower, employing a dynamic high-resolution scheme to manage large-scale satellite imagery.
- Self-Augmentation (Cyclic Referring): To strengthen the link between coordinates and descriptions, the authors use a "cyclic" strategy, converting detection/segmentation data into region captioning tasks and vice versa.
The RSVLTS Benchmark
The paper introduces the Remote Sensing Vision-Language Task Set (RSVLTS), which categorizes tasks into a hierarchy of increasing difficulty: Open-Vocabulary Tasks (OVT), Referring Expression Tasks (RET), and Described Object Tasks (DOT).
Key Insights and Cognitive Hierarchy
A central argument of the work is that RS tasks should be viewed through the lens of cognitive load. Moving from "what is in this image" (OVT) to "where is the object described by X" (RET) to "find the object that meets conditions A, B, and C" (DOT) requires fundamentally different reasoning capabilities. By reducing masks and boxes to "sets of points," the authors avoid task-specific decoders, enabling the LLM to handle spatial coordinates natively. Furthermore, for high-fidelity segmentation, the model integrates with SAM2 (Segment Anything Model 2), using generated points as prompts.
Performance and Versatility
GeoRSMLLM serves as a generalized solution, handling tasks that previously required separate specialized architectures, such as Referring Change Detection (RCD) and Geo-localization. The condition parser significantly improves instruction following, allowing the model to navigate complex spatial queries without the hallucinations typical of standard MLLMs.
Future Directions
While GeoRSMLLM provides a unified framework, the authors note that specialized models still hold an edge in specific areas like change detection. Future work will focus on refining the architecture to better leverage the point-based data and expanding the RSVLTS benchmark to include tasks like object counting and more granular geo-localization.
Summary updated