Abstract
Spectral information has long been recognized as a critical cue in remote sensing observations. Although numerous vision-language models have been developed for pixel-level interpretation, spectral information remains underutilized, resulting in suboptimal performance, particularly in multispectral scenarios. To address this limitation, we construct a vision-language instruction-following dataset named SPIE, which encodes spectral priors of land-cover objects into textual attributes recognizable by large language models (LLMs), based on classical spectral index computations. Leveraging this dataset, we propose SPEX, a multimodal LLM designed for instruction-driven land cover extraction. To this end, we introduce several carefully designed components and training strategies, including multiscale feature aggregation, token context condensation, and multispectral visual pre-training, to achieve precise and flexible pixel-level interpretation. To the best of our knowledge, SPEX is the first multimodal vision-language model dedicated to land cover extraction in spectral remote sensing imagery. Extensive experiments on five public multispectral datasets demonstrate that SPEX consistently outperforms existing state-of-the-art methods in extracting typical land cover categories such as vegetation, buildings, and water bodies. Moreover, SPEX is capable of generating textual explanations for its predictions, thereby enhancing interpretability and user-friendliness.
Introduction
Remote sensing is inherently "more than meets the eye." While humans perceive the world in RGB, satellites capture rich spectral information across infrared bands that are critical for identifying vegetation, water, and urban structures. However, current Vision-Language Models (VLMs) face two major hurdles in this domain: a bias toward natural RGB images (like COCO or ImageNet) and a lack of deep understanding regarding the underlying spectral relationships (e.g., NDVI) that define land cover.
Traditional segmentation models are often "static," meaning they cannot be easily nudged by human instructions or provide explanations for their predictions. SPEX (SPectral instruction EXtraction) bridges this gap by creating a model that is both spectrally aware and interactively flexible, moving beyond simple pixel classification to a more nuanced, instruction-driven understanding of remote sensing imagery.
Methodology
SPEX utilizes a three-pronged approach to integrate spectral intelligence into a multimodal framework:
- The SPIE Dataset: To teach the model about spectral properties, the authors created the Spectral Prompt Instruction Extraction (SPIE) dataset. By using classical spectral indices (NDVI, NDWI, NDBI) to automatically generate textual descriptions of land cover regions, raw spectral data is transformed into a format an LLM can reason with.
- Hierarchical Architecture: The model employs a hierarchical backbone (InternImage-L) for multi-scale feature extraction. A custom Multi-Scale Aggregation Module (MSAM) fuses these features to preserve fine-grained spatial structures, while a Token Compression Projector (TCP) condenses LLM tokens into a compact set of "guidance tokens" for the final mask generator.
- Multi-Stage Training: To bridge the domain gap from natural images, the model undergoes a specific visual pre-training stage on multispectral data before moving to vision-language alignment.
One key innovation is the decoupling of text and mask generation. The segmentation mask is conditioned on the LLM's internal hidden states rather than the final decoded text, which helps prevent textual "hallucinations" from degrading pixel-level accuracy.
Datasets and Training
The researchers curated the SPIE dataset from five prominent public multispectral semantic segmentation benchmarks: SegMunich and Globe230K (vegetation), Chesapeake (high-resolution vegetation), SpaceNet-V2 (buildings), and GID-15 (water bodies). The resulting dataset contains hundreds of thousands of image-instruction-mask triplets, providing a robust foundation for training instruction-following models in the remote sensing domain.
Key Results
SPEX consistently outperformed both traditional segmentation networks (like UperNet) and general-purpose VLMs (like LISA or PixelLM) across multiple categories:
- Vegetation Extraction: On the Globe230K dataset, SPEX achieved an F1 score of 84.6%, outperforming the nearest foundation model (MVPNet) by 4.3%.
- Building & Water Extraction: The model set new state-of-the-art marks on SpaceNet-V2 (F1: 71.6%) and GID-15 (F1: 43.3%), showing particular strength in distinguishing buildings from spectrally similar impervious surfaces.
- Qualitative Performance: Visual results demonstrate that SPEX is significantly better at resolving "boundary adhesion"—the messy edges where different land cover types meet—compared to previous methods.
Conclusion and Future Work
SPEX represents a significant step forward in making remote sensing analysis more interpretable and interactive. By treating spectral priors as textual attributes, it gives LLMs a "spectral cheat sheet" for pixel-level tasks. While currently focused on major categories like vegetation and buildings, future work will involve expanding the category vocabulary and optimizing inference speed to handle massive satellite swaths in real-time.
Summary updated