MSFMamba: Multi-Scale Feature Fusion State Space Model for Multi-Source Remote Sensing Image Classification

Feng Gao, Xuepeng Jin, Xiaowei Zhou, Junyu Dong, Qian Du

MSFMamba: Bringing State Space Efficiency to Multi-Source Remote Sensing

TL;DR

MSFMamba is a novel architecture that applies the efficiency of State Space Models (specifically Mamba) to the joint classification of hyperspectral and LiDAR/SAR data. By introducing multi-scale spatial scanning and a cross-modal fusion mechanism, it achieves state-of-the-art accuracy while using significantly fewer parameters and less computational power than traditional CNNs or Transformers.

What Problem Does This Paper Address?

Classifying remote sensing imagery often requires fusing data from multiple sources, such as Hyperspectral Imaging (HSI), which provides rich spectral detail, and LiDAR or Synthetic Aperture Radar (SAR), which provide structural and elevation data.

The challenge is two-fold:

  1. The Heterogeneity Gap: These data sources have vastly different distributions and structures. Bridging the gap between them is difficult for standard neural networks.
  2. The Complexity vs. Context Trade-off: Convolutional Neural Networks (CNNs) are efficient but have a limited "receptive field" (they can't see the big picture). Transformers offer a global view but suffer from quadratic computational complexity ($O(N^2)$), making them slow and memory-intensive for high-resolution remote sensing tasks.

While the Mamba architecture (based on State Space Models) offers global modeling with linear complexity ($O(N)$), it hasn't been fully optimized for the unique redundancies and multi-modal nature of remote sensing data.

Methods

The authors propose MSFMamba, which consists of three specialized blocks:

  • Multi-Scale Spatial Mamba (MSpa-Mamba): Standard Mamba models scan images in multiple directions (e.g., forward, backward, horizontal, vertical) to capture spatial context. This creates massive feature redundancy. MSpa-Mamba uses a multi-scale strategy, downsampling some scanning routes to reduce computational load while maintaining a large effective receptive field.
  • Spectral Mamba (Spe-Mamba): This block treats the spectral bands of HSI as a sequence. It reshapes the data to focus specifically on the relationships between different wavelengths, which is crucial for identifying materials on the ground.
  • Fusion Mamba (Fus-Mamba): This is the core innovation for multi-source data. Instead of simply concatenating features, the Fus-SSM mechanism derives the state transition parameters for one modality (e.g., HSI) from the other modality (e.g., LiDAR). This forces the model to learn deep, cross-modal interactions during the scanning process.

Datasets

The model was evaluated on four diverse, real-world benchmark datasets:

  • Berlin: HSI and SAR data covering urban and rural areas.
  • Augsburg: HSI and SAR data with a focus on land cover.
  • Houston 2018: HSI and multispectral LiDAR data.
  • Houston 2013: HSI and LiDAR data from the 2013 Data Fusion Contest.

Key Insights and Ideas

  • Efficiency is the "Aha" Moment: The most striking insight is the efficiency gain. MSFMamba achieves better results than the Transformer-based ExViT while using roughly 1/3 of the parameters and 1/10 of the FLOPs (Floating Point Operations).
  • Cross-Modal Parameter Generation: In the Fusion Mamba block, the model doesn't just "look" at both sources; it uses the features of one source to define how the other source's information should flow through the state space. This is a more integrated approach than standard cross-attention.
  • Superior Receptive Field: Visualization of the "Effective Receptive Field" shows that MSFMamba captures a much broader context than CNN-based models, which explains its superior performance in identifying large-scale urban structures.

Key Results

MSFMamba outperformed several state-of-the-art (SOTA) models, including CNN-Transformer hybrids and other Mamba variants:

  • Overall Accuracy (OA): Achieved 76.92% (Berlin), 91.38% (Augsburg), 92.38% (Houston 2018), and 92.86% (Houston 2013).
  • Model Size: On the Augsburg dataset, MSFMamba used only 1.52M parameters, compared to 37.7M for FusAtNet and 4.78M for HCT.
  • Inference Speed: It demonstrated the lowest inference time (0.17s on Augsburg), making it highly suitable for deployment on resource-constrained edge devices like drones or satellites.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Imbalanced Classes: Like most deep learning models, accuracy drops on classes with very few training samples.
  • Interpretability: While the results are strong, the internal "logic" of State Space Models is less intuitive than the attention maps provided by Transformers.
  • Future Directions: The team plans to explore dynamic sampling techniques and hybrid architectures that combine Mamba with Transformer elements to further enhance long-range feature modeling.

Source Code: The authors have made their code available at https://github.com/oucailab/MSFMamba.

Summary updated

Abstract

In the field of multi-source remote sensing image classification, remarkable progress has been made by using Convolutional Neural Network (CNN) and Transformer. Recently, Mamba-based methods built upon the State Space Model (SSM) have shown great potential for long-range dependency modeling with linear complexity, but they have rarely been explored for multi-source remote sensing image classification tasks. To address this issue, we propose the Multi-Scale Feature Fusion Mamba (MSFMamba) network, a novel framework designed for the joint classification of hyperspectral image (HSI) and Light Detection and Ranging (LiDAR)/Synthetic Aperture Radar (SAR) data. The MSFMamba network is composed of three key components: the Multi-Scale Spatial Mamba (MSpa-Mamba) block, the Spectral Mamba (Spe-Mamba) block, and the Fusion Mamba (Fus-Mamba) block. The MSpa-Mamba block employs a multi-scale strategy to reduce computational cost and alleviate feature redundancy in multiple scanning routes, ensuring efficient spatial feature modeling. The Spe-Mamba block focuses on spectral feature extraction, addressing the unique challenges of HSI data representation. Finally, the Fus-Mamba block bridges the heterogeneous gap between HSI and LiDAR/SAR data by extending the original Mamba architecture to accommodate dual inputs, enhancing cross-modal feature interactions and enabling seamless data fusion. Together, these components enable MSFMamba to effectively tackle the challenges of multi-source data classification, delivering improved performance with optimized computational efficiency. Comprehensive experiments on four real-world multi-source remote sensing datasets demonstrate the superiority of MSFMamba outperforms several state-of-the-art methods. The source codes of MSFMamba are publicly available at https://github.com/oucailab/MSFMamba.
BibTeX
@article{gao2025msfmamba,
  author = {Feng Gao and Xuepeng Jin and Xiaowei Zhou and Junyu Dong and Qian Du},
  title = {{MSFMamba: Multi-Scale Feature Fusion State Space Model for Multi-Source Remote Sensing Image Classification}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2025},
  doi = {10.1109/tgrs.2025.3535622},
}
Figure