Dual-Branch Subpixel-Guided Network for Hyperspectral Image Classification

Zhu Han, Jin Yang, Lianru Gao, Zhiqiang Zeng, Bing Zhang, Jocelyn Chanussot

TL;DR

This paper introduces DSNet, a novel dual-branch deep learning network for hyperspectral image (HSI) classification. DSNet uniquely integrates subpixel information, extracted via an unsupervised autoencoder unmixing architecture, with traditional pixel-level spectral-spatial features. This fusion significantly improves classification accuracy, especially in scenarios with mixed pixels due to low spatial resolution.

What Problem Does This Paper Address?

Hyperspectral imaging (HSI) captures detailed spectral information, enabling the identification of various materials. However, classifying these images is challenging, particularly when sensor resolution leads to "mixed pixels" – pixels that contain multiple ground objects. Existing deep learning (DL) methods often focus on complex pixel-level spectral and spatial feature extraction but overlook the subpixel information that describes the spatial distribution of materials within a single pixel. This oversight can lead to misclassifications. While hyperspectral unmixing (HU) techniques address subpixel information, they are often treated independently from classification. This paper aims to bridge this gap by developing a unified framework that leverages both subpixel and pixel-level information for more robust and accurate HSI classification.

Methods

The core of DSNet is a dual-branch architecture:

  1. Deep Autoencoder (AE) Unmixing Network: This unsupervised branch uses a deep AE to extract subpixel-level abundance information. It's designed with a general mixing decoder that accounts for both linear and physically nonlinear mixture mechanisms. This allows it to learn diagnostic abundances without requiring prior knowledge of endmembers or mixing models. The AE's encoder maps input patches to high-dimensional representations, and the decoder reconstructs them, enabling the extraction of abundance maps.
  2. CNN-based Classifier Network: This branch extracts spectral-spatial features from HSI patches using standard 2D convolutional layers. It generates pixel-level class features that contribute to the final classification.
  3. Subpixel Fusion Module: This crucial component fuses the outputs from the AE unmixing network (abundance maps) and the CNN classifier (class features). It ensures consistency in spatial dimensions and concatenates the information, preserving as much detail as possible. This fused representation is then fed into a linear layer for the final class prediction.

The network is trained using a combined loss function: a reconstruction loss (RE) for the AE unmixing branch and a cross-entropy loss (CE) for the classification branch.

Datasets

The proposed DSNet was evaluated on three widely used HSI benchmark datasets:

  • Indian Pines: Captured by AVIRIS, it has 220 spectral bands (reduced to 200 after preprocessing) and 16 land cover classes.
  • Berlin: Collected by EnMAP, it features 244 spectral bands and 8 land cover classes.
  • Augsburg: Acquired by HySpex, it has 180 spectral bands and 7 land cover classes.

Key Insights and Ideas

  • Subpixel Information is Crucial: The paper strongly advocates for the importance of subpixel information, especially in low-resolution HSI data, to overcome the limitations of purely pixel-based classification.
  • Unsupervised Unmixing for Classification: Integrating an unsupervised AE unmixing network directly into the classification pipeline allows DSNet to learn valuable subpixel features without the need for labeled unmixing data, making it more practical.
  • Physically Nonlinear Mixing Matters: The inclusion of a nonlinear mixing model in the AE decoder is shown to be more effective than a purely linear model, better capturing the complex spectral mixing phenomena in real-world scenarios.
  • Effective Fusion Strategy: The subpixel fusion module is key to successfully combining the distinct information from the unmixing and classification branches, leading to improved performance.
  • Robustness to Mixed Pixels: DSNet demonstrates superior performance in handling mixed pixels, a common challenge in HSI classification, by leveraging its subpixel understanding.

Key Results

  • State-of-the-Art Performance: DSNet consistently outperformed several classic and hybrid backbone networks (including 2-D CNN, 3-D CNN, GRU, ViT, MorphConv, SSFTT, and WFCG) across all three benchmark datasets.
  • Accuracy Improvements: On the Indian Pines dataset, DSNet achieved an Overall Accuracy (OA) of 93.08%, compared to the next best hybrid method (WFCG) at 90.64%. Similar significant gains were observed on the Berlin (OA: 72.09% vs. 68.83%) and Augsburg (OA: 89.30% vs. 87.66%) datasets.
  • Visual Superiority: Qualitative results (classification maps) showed that DSNet produced more accurate visualizations and preserved finer spatial structures, especially for challenging classes.
  • Computational Efficiency: While not the fastest, DSNet's computational cost was deemed acceptable for HSI classification tasks, offering a good trade-off between performance and efficiency. The execution time for one epoch was competitive, especially compared to some complex hybrid models.
  • Parameter Sensitivity: The number of decoder layers in the AE had a relatively stable impact on performance, with 2-4 layers being optimal. The fusion coefficient ($\lambda$) showed some sensitivity, with optimal values varying across datasets, but generally, a small to moderate value (e.g., 0.2-0.8) provided the best results, indicating the importance of balancing both branches.

Limitations and Future Work

  • Training Data Requirement: The paper notes that DSNet, particularly its unmixing component, requires a certain number of training samples to reliably extract subpixel information. Performance can be slightly lower than some hybrid methods at very low training ratios (e.g., 20%).
  • Computational Cost: While acceptable, DSNet's dual-branch architecture is more computationally intensive than simpler single-branch networks.
  • Future Directions: The authors suggest integrating more advanced spectral-spatial classifiers, such as hybrid CNN-Transformer architectures, into the AE network to further enhance performance. They also aim to explore more complete spectral-spatial classifier integration.

Summary updated

Abstract

Deep learning (DL) has been widely applied to hyperspectral image (HSI) classification, owing to its promising feature learning and representation capabilities. However, limited by the spatial resolution of sensors, existing DL-based classification approaches mainly focus on pixel-level spectral and spatial information extraction through complex network architecture design while ignoring the existence of mixed pixels in actual scenarios. To tackle this difficulty, we propose a novel dual-branch subpixel-guided network for HSI classification, called DSNet, which automatically integrates subpixel information and convolutional class features by introducing a deep autoencoder unmixing architecture to enhance classification performance. DSNet is capable of fully considering physically nonlinear properties within subpixels and adaptively generating diagnostic abundances in an unsupervised manner to achieve more reliable decision boundaries for class label distributions. The subpixel fusion module is designed to ensure high-quality information fusion across pixel and subpixel features, further promoting stable joint classification. Experimental results on three benchmark datasets demonstrate the effectiveness and superiority of DSNet compared with state-of-the-art DL-based HSI classification approaches. The codes will be available at https://github.com/hanzhu97702/DSNet , contributing to the remote sensing community.
BibTeX
@article{han2024dual,
  author = {Zhu Han and Jin Yang and Lianru Gao and Zhiqiang Zeng and Bing Zhang and Jocelyn Chanussot},
  title = {{Dual-Branch Subpixel-Guided Network for Hyperspectral Image Classification}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  url = {https://openreview.net/forum?id=Qh4rnEhyjP},
}
Figure