TL;DR
This paper introduces DBMamba, a novel deep learning architecture for hyperspectral image (HSI) classification. By combining a dual-branch CNN for initial feature extraction with a bidirectional Mamba for capturing long-range spectral dependencies, DBMamba achieves state-of-the-art performance while addressing the limitations of traditional CNNs and Transformers in handling sequential spectral data.
What Problem Does This Paper Address?
Hyperspectral imaging captures detailed spectral information across hundreds of narrow bands, making it invaluable for applications like environmental monitoring and land cover classification. However, classifying HSI pixels accurately is challenging due to factors like atmospheric interference and the "same spectrum, different materials" phenomenon. While Convolutional Neural Networks (CNNs) have been successful in HSI classification, they struggle to effectively model the sequential nature of spectral features and can become computationally expensive with increased depth. Transformers, while good at capturing long-range dependencies, can be prone to noise and outliers in HSI data and suffer from quadratic computational complexity with sequence length. This paper aims to bridge this gap by developing a model that efficiently leverages both spectral-spatial features and long-range spectral dependencies without the computational burden or noise sensitivity of Transformers.
Methods
The proposed DBMamba architecture consists of three main components:
- CNN-Based Dual-Branch Feature Extraction: This module first uses Principal Component Analysis (PCA) to reduce the spectral dimensionality of the HSI data. Then, a dual-branch CNN is employed:
- Branch 1 (3D-CNN): Extracts joint spectral-spatial features from 3D blocks of the HSI data.
-
Branch 2 (2D-CNN): Extracts spatial features from 2D representations of the HSI data. The features from these two branches are fused channel-wise.
-
HSI Feature Tokenization: The fused features are then tokenized. This involves flattening the features and applying Gaussian weighting to emphasize important features, creating "semantic tokens" that represent high-level concepts.
- Bidirectional Mamba (Bim) Encoder Module: This is the core innovation. It takes the semantic tokens and:
- Adds a learnable class token and positional embeddings to capture global context and positional information.
- Employs a bidirectional Mamba structure. Unlike standard Mamba or Transformers, this bidirectional approach processes the token sequence in both forward and backward directions. This allows it to capture long-range dependencies more effectively and robustly, mitigating issues with noise and outliers.
- Finally, an MLP head uses the output to produce the classification results.
Datasets
The DBMamba model was evaluated on three widely used hyperspectral image datasets:
- Indian Pines: Collected in Indiana, USA, with 224 spectral bands (reduced to 200). It has a 145x145 pixel resolution and 16 land cover classes.
- Pavia University: Collected in Pavia, Italy, with 115 spectral bands (reduced to 103). It has a 610x340 pixel resolution and 9 land cover classes.
- Salinas: Collected in Salinas Valley, California, USA, with 224 spectral bands (reduced to 204). It has a 512x217 pixel resolution and 16 land cover classes.
Training samples were randomly selected, with specific percentages (e.g., 10% for Indian Pines) noted for experiments.
Key Insights and Ideas
- Bridging CNNs and Mamba for HSI: The paper successfully integrates the local feature extraction strengths of CNNs with the sequential modeling capabilities of Mamba, creating a powerful hybrid architecture.
- Bidirectional Mamba for Robustness: The key innovation is the bidirectional Mamba. By processing sequences in both directions, it gains a more comprehensive understanding of spectral context and is less susceptible to noise and outliers, a common problem in HSI data. This addresses a limitation of unidirectional Mamba and standard Transformers.
- Dual-Branch CNN for Comprehensive Shallow Features: Using separate branches for spectral-spatial and pure spatial features, then fusing them, ensures that rich, low-level information is captured before being passed to the Mamba encoder.
- Tokenization for Feature Emphasis: The feature tokenization step, with Gaussian weighting, helps the model focus on the most discriminative features, improving overall classification.
Key Results
DBMamba demonstrated superior performance across all three datasets compared to several state-of-the-art methods, including SVM, 1D-CNN, 2D-CNN, 3D-CNN, HybridSN, and SSFTT.
- Indian Pines: Achieved an Overall Accuracy (OA) of 98.72%, outperforming SSFTT by 1.04%.
- Pavia University: Achieved an OA of 99.40%, outperforming SSFTT by 0.09%.
- Salinas: Achieved an OA of 99.97%, outperforming SSFTT by 0.15%.
The paper also conducted ablation studies, showing that removing the bidirectional Mamba component led to a significant drop in OA (5.25%), highlighting its critical role. Parameter analysis indicated that an input window size of 28x28 and a batch size of 64 were optimal for most datasets.
Limitations and Future Work
- Model Complexity: While DBMamba outperforms other methods, the authors acknowledge that there is room for improvement in terms of model complexity, particularly in reducing the number of parameters to shorten runtime.
- Computational Cost: Despite Mamba's linear complexity, the use of a larger input window size (28x28) and the overall architecture still result in a longer training time compared to some simpler methods.
- Future Directions: The authors suggest future work on designing end-to-end lightweight Mamba architectures, exploring cross-scanning with Mamba for improved HSI classification, and leveraging Mamba's strengths for joint fusion of hyperspectral and lidar data.
Summary updated