MSFMamba: Multiscale Feature Fusion State Space Model for Multisource Remote Sensing Image Classification

Feng Gao, Xuepeng Jin, Xiaowei Zhou, Junyu Dong, Qian Du

TL;DR

This paper introduces MSFMamba, a novel deep learning architecture that leverages the efficiency of State Space Models (SSMs) for multi-source remote sensing image classification. By effectively fusing hyperspectral (HSI) and SAR/LiDAR data through multi-scale spatial and spectral feature extraction, MSFMamba achieves state-of-the-art performance with improved computational efficiency.

What Problem Does This Paper Address?

Classifying remote sensing images from multiple sources (like hyperspectral, LiDAR, and SAR) is crucial for applications ranging from land use mapping to urban planning. However, these data sources have fundamentally different characteristics (e.g., spectral richness vs. spatial detail), creating a "heterogeneity gap" that is challenging to bridge. While Convolutional Neural Networks (CNNs) and Transformers have shown promise, CNNs struggle with long-range dependencies, and Transformers are computationally expensive. Recently, Mamba, based on State Space Models (SSMs), has emerged as an efficient alternative for modeling long-range dependencies with linear complexity. However, its application to multi-source remote sensing classification, particularly in fusing heterogeneous data, remains underexplored. This paper aims to address this gap by proposing an SSM-based framework that can effectively fuse multi-source data while mitigating issues like feature redundancy and computational cost.

Methods

The core of MSFMamba is its novel Spatial-Spectral Mamba Module, which is designed to handle the complexities of multi-source data. This module comprises three key components:

  • Multi-Scale Spatial Mamba (MSpa-Mamba) block: This block addresses feature redundancy in SSMs by employing a multi-scale strategy. It uses depth-wise convolutions with different strides to capture spatial features at various granularities, reducing computational burden and improving efficiency.
  • Spectral Mamba (Spe-Mamba) block: Specifically designed for Hyperspectral Imagery (HSI), this block focuses on extracting rich spectral features, tackling the unique challenges of HSI data representation.
  • Fusion Mamba (Fus-Mamba) block: This is the crucial component for bridging the heterogeneity gap. It extends the original Mamba architecture to accept dual inputs (from HSI and LiDAR/SAR), enabling enhanced cross-modal feature interaction and seamless data fusion. The Fusion SSM (Fus-SSM) within this block is designed symmetrically, allowing each modality's features to inform the projection and timescale parameters of the other, leading to deeper inter-modal understanding.

The overall framework first preprocesses HSI using PCA, then feeds both HSI and LiDAR/SAR data through multiple stacked Spatial-Spectral Mamba Modules for feature extraction and fusion, culminating in a classifier for land cover prediction.

Datasets

The proposed MSFMamba was evaluated on four publicly available multi-source remote sensing datasets:

  • Berlin Dataset: Hyperspectral (simulated EnMAP) and SAR (Sentinel-1) data.
  • Augsburg Dataset: Hyperspectral (HySpex) and SAR (Sentinel-1) data.
  • Houston2018 Dataset: Hyperspectral and multispectral LiDAR data.
  • Houston2013 Dataset: Hyperspectral and LiDAR data.

These datasets cover urban and rural areas and are commonly used benchmarks for multi-source remote sensing classification.

Key Insights and Ideas

  • SSMs as a powerful alternative to Transformers: The paper demonstrates that SSMs, with their linear complexity and ability to model long-range dependencies, can be highly effective for remote sensing image classification, offering a more computationally efficient alternative to Transformers.
  • Multi-scale spatial processing is key for SSM efficiency: By incorporating a multi-scale strategy in the MSpa-Mamba block, the authors effectively reduce feature redundancy inherent in multi-scan SSM approaches, leading to better performance and efficiency.
  • Dedicated spectral processing for HSI is vital: The Spe-Mamba block highlights the importance of specialized modules for extracting spectral information from HSI, which is crucial for accurate classification.
  • Symmetric cross-modal fusion unlocks deeper understanding: The Fus-Mamba block's symmetric design, where each modality's features inform the other's SSM parameters, is a novel approach to effectively bridge the heterogeneity gap between different data types.

Key Results

MSFMamba achieved state-of-the-art performance across all four datasets:

  • Berlin Dataset: Overall Accuracy (OA) of 76.92%, Average Accuracy (AA) of 64.88%, and Kappa coefficient of 64.88%.
  • Augsburg Dataset: OA of 91.38%, AA of 92.86%.
  • Houston2018 Dataset: OA of 92.38%.
  • Houston2013 Dataset: OA of 92.86%.

Qualitatively, MSFMamba produced significantly cleaner and more accurate classification maps, with reduced noise and better distinction of fine-grained classes, especially in challenging areas. The paper also shows that MSFMamba has lower FLOPs and inference time compared to other state-of-the-art methods, indicating its computational efficiency.

Limitations and Future Work

The authors acknowledge that the model has limitations, including reduced accuracy for imbalanced classes and limited interpretability, which are critical for applications like urban planning and disaster management. Future work will focus on:

  • Developing hybrid models that combine Mamba with Transformer architectures for even better long-range feature modeling and multi-modal fusion.
  • Improving the interpretability of MSFMamba, particularly in understanding how it processes and integrates contributions from different data modalities.
  • Exploring feature importance and feature selection techniques to further enhance efficiency and transparency.

Summary updated

Abstract

In the field of multi-source remote sensing image classification, remarkable progress has been made by using Convolutional Neural Network (CNN) and Transformer. Recently, Mamba-based methods built upon the State Space Model (SSM) have shown great potential for long-range dependency modeling with linear complexity, but they have rarely been explored for multi-source remote sensing image classification tasks. To address this issue, we propose the Multi-Scale Feature Fusion Mamba (MSFMamba) network, a novel framework designed for the joint classification of hyperspectral image (HSI) and Light Detection and Ranging (LiDAR)/Synthetic Aperture Radar (SAR) data. The MSFMamba network is composed of three key components: the Multi-Scale Spatial Mamba (MSpa-Mamba) block, the Spectral Mamba (Spe-Mamba) block, and the Fusion Mamba (Fus-Mamba) block. The MSpa-Mamba block employs a multi-scale strategy to reduce computational cost and alleviate feature redundancy in multiple scanning routes, ensuring efficient spatial feature modeling. The Spe-Mamba block focuses on spectral feature extraction, addressing the unique challenges of HSI data representation. Finally, the Fus-Mamba block bridges the heterogeneous gap between HSI and LiDAR/SAR data by extending the original Mamba architecture to accommodate dual inputs, enhancing cross-modal feature interactions and enabling seamless data fusion. Together, these components enable MSFMamba to effectively tackle the challenges of multi-source data classification, delivering improved performance with optimized computational efficiency. Comprehensive experiments on four real-world multi-source remote sensing datasets demonstrate the superiority of MSFMamba outperforms several state-of-the-art methods. The source codes of MSFMamba are publicly available at https://github.com/oucailab/MSFMamba.
BibTeX
@article{gao2025msfmamba,
  author = {Feng Gao and Xuepeng Jin and Xiaowei Zhou and Junyu Dong and Qian Du},
  title = {{MSFMamba: Multiscale Feature Fusion State Space Model for Multisource Remote Sensing Image Classification}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2025},
  doi = {10.1109/tgrs.2025.3535622},
}
Figure