Abstract
Remote sensing (RS) image classification is hindered by sensor heterogeneity and geographic domain shifts. Traditional models often fail when applied to unseen environments or different sensor modalities. This paper introduces the Frequency-Aware Vision-Language Multimodality Generalization Network (FVMGN), a novel framework for the RS Multimodality Generalization (RSMG) task. FVMGN utilizes wavelet-based frequency disentanglement to extract domain-invariant structural features and integrates modality-specific linguistic priors generated by Large Language Models (LLMs) to provide physical context for diverse sensors. Evaluated on cross-domain tasks involving Hyperspectral and LiDAR data, FVMGN achieves a state-of-the-art Overall Accuracy of 93.19%, demonstrating robust generalization capabilities across diverse geographic scenes.
Introduction
The proliferation of remote sensing (RS) technologies has introduced a variety of sensors, including Hyperspectral (HS), LiDAR, and SAR, each capturing unique physical properties. However, this diversity leads to data heterogeneity. Furthermore, models trained on specific geographic regions often suffer from domain shift when deployed elsewhere due to variations in architecture, vegetation, and sensor characteristics. The RS Multimodality Generalization (RSMG) task aims to address these issues by training models on one multimodal scene that can generalize to entirely unseen multimodal environments.
Methodology
The FVMGN architecture is designed to extract domain-invariant features through a multi-stage pipeline that bridges the gap between vision and language.
Core Components
- Diffusion-based Augmentation (DTAug): To combat limited data diversity, a Denoising Diffusion Probabilistic Model (DDPM) is employed to reconstruct multimodal land-cover distributions, enriching the training set before feature extraction.
- Multimodal Wavelet Disentanglement (MWDis): This module applies wavelet transforms to separate images into low-frequency (style-specific) and high-frequency (structural) components. By focusing on high-frequency structural edges, the model learns to ignore domain-specific noise and "style" variations between datasets.
- Spatial-Frequency-aware Image Encoder (SFIE): A hybrid encoder combining CNN and Vision Transformer (ViT) architectures captures local and global features across both spatial and frequency domains.
- Modality-Specific Text Priors: Standard class labels are replaced with detailed "proprietary" text descriptions generated by LLMs. These descriptions reflect the specific physical attributes captured by different sensors, such as spectral textures for HS or elevation profiles for LiDAR.
- Multiscale Spatial-Frequency Feature Alignment (MSFFA): This module aligns vision and language features across multiple scales, ensuring a unified semantic space for robust classification.
Experiments and Results
FVMGN was evaluated on three major multimodal RS datasets: MUUFL (Gulfport), Trento, and Houston 2013 (HU2013). The experiments focused on cross-domain tasks, such as training on MUUFL and testing on Houston, which is significantly more challenging than standard intra-dataset classification.
FVMGN consistently outperformed state-of-the-art (SOTA) methods across all dataset combinations. In the MUUFL to Houston task, the model achieved an Overall Accuracy (OA) of 93.19%, a substantial improvement over previous methods that typically achieved between 40% and 70%. Qualitative analysis showed that FVMGN produces cleaner classification maps with significantly less noise, and the model remained robust even when training data was limited.
Conclusion
FVMGN demonstrates that frequency-domain analysis is a powerful tool for disentangling domain-invariant features in remote sensing. By combining this with "linguistic physics"—modality-specific text priors—the framework successfully bridges the gap between heterogeneous sensors and diverse geographic domains. While the current pipeline requires some manual refinement of text descriptions, future work aims to automate this process and optimize the system's computational efficiency for real-time applications.
Summary updated