Bidirectional Mamba with Dual-Branch Feature Extraction for Hyperspectral Image Classification

Ming Sun, Jie Zhang, Xin-Chen He, Yanfei ZhongSensors 2024 · DOI

TL;DR

This paper introduces DBMamba, a novel deep learning architecture for hyperspectral image (HSI) classification. By combining a dual-branch CNN for initial feature extraction with a bidirectional Mamba for capturing long-range spectral dependencies, DBMamba achieves state-of-the-art performance while addressing the limitations of traditional CNNs and Transformers in handling sequential spectral data.

What Problem Does This Paper Address?

Hyperspectral imaging captures detailed spectral information across hundreds of narrow bands, making it invaluable for applications like environmental monitoring and land cover classification. However, classifying HSI pixels accurately is challenging due to factors like atmospheric interference and the "same spectrum, different materials" phenomenon. While Convolutional Neural Networks (CNNs) have been successful in HSI classification, they struggle to effectively model the sequential nature of spectral features and can become computationally expensive with increased depth. Transformers, while good at capturing long-range dependencies, can be prone to noise and outliers in HSI data and suffer from quadratic computational complexity with sequence length. This paper aims to bridge this gap by developing a model that efficiently leverages both spectral-spatial features and long-range spectral dependencies without the computational burden or noise sensitivity of Transformers.

Methods

The proposed DBMamba architecture consists of three main components:

  1. CNN-Based Dual-Branch Feature Extraction: This module first uses Principal Component Analysis (PCA) to reduce the spectral dimensionality of the HSI data. Then, a dual-branch CNN is employed:
  2. Branch 1 (3D-CNN): Extracts joint spectral-spatial features from 3D blocks of the HSI data.
  3. Branch 2 (2D-CNN): Extracts spatial features from 2D representations of the HSI data. The features from these two branches are fused channel-wise.

  4. HSI Feature Tokenization: The fused features are then tokenized. This involves flattening the features and applying Gaussian weighting to emphasize important features, creating "semantic tokens" that represent high-level concepts.

  5. Bidirectional Mamba (Bim) Encoder Module: This is the core innovation. It takes the semantic tokens and:
  6. Adds a learnable class token and positional embeddings to capture global context and positional information.
  7. Employs a bidirectional Mamba structure. Unlike standard Mamba or Transformers, this bidirectional approach processes the token sequence in both forward and backward directions. This allows it to capture long-range dependencies more effectively and robustly, mitigating issues with noise and outliers.
  8. Finally, an MLP head uses the output to produce the classification results.

Datasets

The DBMamba model was evaluated on three widely used hyperspectral image datasets:

  • Indian Pines: Collected in Indiana, USA, with 224 spectral bands (reduced to 200). It has a 145x145 pixel resolution and 16 land cover classes.
  • Pavia University: Collected in Pavia, Italy, with 115 spectral bands (reduced to 103). It has a 610x340 pixel resolution and 9 land cover classes.
  • Salinas: Collected in Salinas Valley, California, USA, with 224 spectral bands (reduced to 204). It has a 512x217 pixel resolution and 16 land cover classes.

Training samples were randomly selected, with specific percentages (e.g., 10% for Indian Pines) noted for experiments.

Key Insights and Ideas

  • Bridging CNNs and Mamba for HSI: The paper successfully integrates the local feature extraction strengths of CNNs with the sequential modeling capabilities of Mamba, creating a powerful hybrid architecture.
  • Bidirectional Mamba for Robustness: The key innovation is the bidirectional Mamba. By processing sequences in both directions, it gains a more comprehensive understanding of spectral context and is less susceptible to noise and outliers, a common problem in HSI data. This addresses a limitation of unidirectional Mamba and standard Transformers.
  • Dual-Branch CNN for Comprehensive Shallow Features: Using separate branches for spectral-spatial and pure spatial features, then fusing them, ensures that rich, low-level information is captured before being passed to the Mamba encoder.
  • Tokenization for Feature Emphasis: The feature tokenization step, with Gaussian weighting, helps the model focus on the most discriminative features, improving overall classification.

Key Results

DBMamba demonstrated superior performance across all three datasets compared to several state-of-the-art methods, including SVM, 1D-CNN, 2D-CNN, 3D-CNN, HybridSN, and SSFTT.

  • Indian Pines: Achieved an Overall Accuracy (OA) of 98.72%, outperforming SSFTT by 1.04%.
  • Pavia University: Achieved an OA of 99.40%, outperforming SSFTT by 0.09%.
  • Salinas: Achieved an OA of 99.97%, outperforming SSFTT by 0.15%.

The paper also conducted ablation studies, showing that removing the bidirectional Mamba component led to a significant drop in OA (5.25%), highlighting its critical role. Parameter analysis indicated that an input window size of 28x28 and a batch size of 64 were optimal for most datasets.

Limitations and Future Work

  • Model Complexity: While DBMamba outperforms other methods, the authors acknowledge that there is room for improvement in terms of model complexity, particularly in reducing the number of parameters to shorten runtime.
  • Computational Cost: Despite Mamba's linear complexity, the use of a larger input window size (28x28) and the overall architecture still result in a longer training time compared to some simpler methods.
  • Future Directions: The authors suggest future work on designing end-to-end lightweight Mamba architectures, exploring cross-scanning with Mamba for improved HSI classification, and leveraging Mamba's strengths for joint fusion of hyperspectral and lidar data.

Summary updated

Abstract

The hyperspectral image (HSI) classification task is widely used in remote sensing image analysis. The HSI classification methods based on convolutional neural networks (CNNs) have greatly improved the classification performance. However, they cannot well utilize the sequential properties of spectral features and face the challenge of increasing computational cost with the increase in network depth. To address these shortcomings, this paper proposes a novel network with a CNN-Mamba architecture, called DBMamba, which uses a bidirectional Mamba to process spectral feature sequences at a linear computational cost. In the DBMamba, principal component analysis (PCA) is first used to extract the main features of the data. Then, a dual-branch CNN structure, with the fused features from spectral-spatial features by 3D-CNN and spatial features by 2D-CNN, is used to extract shallow spectral-spatial features. Finally, a bidirectional Mamba is used to effectively capture global contextual information in features and significantly enhance the extraction of spectral features. Experimental results on the Indian Pines, Salinas, and Pavia University datasets demonstrate that the classification performance surpasses that of many cutting-edge methods, improving by 1.04%, 0.15%, and 0.09%, respectively, over the competing SSFTT method. The research in this paper enhances the existing knowledge on HSI classification and provides valuable insights for future research in this field.
BibTeX
@inproceedings{sun2024bidirectional,
  author = {Ming Sun and Jie Zhang and Xin-Chen He and Yanfei Zhong},
  title = {{Bidirectional Mamba with Dual-Branch Feature Extraction for Hyperspectral Image Classification}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24216899},
}
Figure