MSFMamba: Bringing State Space Efficiency to Multi-Source Remote Sensing
TL;DR
MSFMamba is a novel architecture that applies the efficiency of State Space Models (specifically Mamba) to the joint classification of hyperspectral and LiDAR/SAR data. By introducing multi-scale spatial scanning and a cross-modal fusion mechanism, it achieves state-of-the-art accuracy while using significantly fewer parameters and less computational power than traditional CNNs or Transformers.
What Problem Does This Paper Address?
Classifying remote sensing imagery often requires fusing data from multiple sources, such as Hyperspectral Imaging (HSI), which provides rich spectral detail, and LiDAR or Synthetic Aperture Radar (SAR), which provide structural and elevation data.
The challenge is two-fold:
- The Heterogeneity Gap: These data sources have vastly different distributions and structures. Bridging the gap between them is difficult for standard neural networks.
- The Complexity vs. Context Trade-off: Convolutional Neural Networks (CNNs) are efficient but have a limited "receptive field" (they can't see the big picture). Transformers offer a global view but suffer from quadratic computational complexity ($O(N^2)$), making them slow and memory-intensive for high-resolution remote sensing tasks.
While the Mamba architecture (based on State Space Models) offers global modeling with linear complexity ($O(N)$), it hasn't been fully optimized for the unique redundancies and multi-modal nature of remote sensing data.
Methods
The authors propose MSFMamba, which consists of three specialized blocks:
- Multi-Scale Spatial Mamba (MSpa-Mamba): Standard Mamba models scan images in multiple directions (e.g., forward, backward, horizontal, vertical) to capture spatial context. This creates massive feature redundancy. MSpa-Mamba uses a multi-scale strategy, downsampling some scanning routes to reduce computational load while maintaining a large effective receptive field.
- Spectral Mamba (Spe-Mamba): This block treats the spectral bands of HSI as a sequence. It reshapes the data to focus specifically on the relationships between different wavelengths, which is crucial for identifying materials on the ground.
- Fusion Mamba (Fus-Mamba): This is the core innovation for multi-source data. Instead of simply concatenating features, the Fus-SSM mechanism derives the state transition parameters for one modality (e.g., HSI) from the other modality (e.g., LiDAR). This forces the model to learn deep, cross-modal interactions during the scanning process.
Datasets
The model was evaluated on four diverse, real-world benchmark datasets:
- Berlin: HSI and SAR data covering urban and rural areas.
- Augsburg: HSI and SAR data with a focus on land cover.
- Houston 2018: HSI and multispectral LiDAR data.
- Houston 2013: HSI and LiDAR data from the 2013 Data Fusion Contest.
Key Insights and Ideas
- Efficiency is the "Aha" Moment: The most striking insight is the efficiency gain. MSFMamba achieves better results than the Transformer-based ExViT while using roughly 1/3 of the parameters and 1/10 of the FLOPs (Floating Point Operations).
- Cross-Modal Parameter Generation: In the Fusion Mamba block, the model doesn't just "look" at both sources; it uses the features of one source to define how the other source's information should flow through the state space. This is a more integrated approach than standard cross-attention.
- Superior Receptive Field: Visualization of the "Effective Receptive Field" shows that MSFMamba captures a much broader context than CNN-based models, which explains its superior performance in identifying large-scale urban structures.
Key Results
MSFMamba outperformed several state-of-the-art (SOTA) models, including CNN-Transformer hybrids and other Mamba variants:
- Overall Accuracy (OA): Achieved 76.92% (Berlin), 91.38% (Augsburg), 92.38% (Houston 2018), and 92.86% (Houston 2013).
- Model Size: On the Augsburg dataset, MSFMamba used only 1.52M parameters, compared to 37.7M for FusAtNet and 4.78M for HCT.
- Inference Speed: It demonstrated the lowest inference time (0.17s on Augsburg), making it highly suitable for deployment on resource-constrained edge devices like drones or satellites.
Limitations and Future Work
The authors acknowledge a few areas for improvement:
- Imbalanced Classes: Like most deep learning models, accuracy drops on classes with very few training samples.
- Interpretability: While the results are strong, the internal "logic" of State Space Models is less intuitive than the attention maps provided by Transformers.
- Future Directions: The team plans to explore dynamic sampling techniques and hybrid architectures that combine Mamba with Transformer elements to further enhance long-range feature modeling.
Source Code: The authors have made their code available at https://github.com/oucailab/MSFMamba.
Summary updated