Transformer based Endmember Fusion with Spatial Context for Hyperspectral Unmixing

H. M. K. D. Wickramathilaka, Rukmal Ratnayake, D. M. U. P. Sumanasekara, Roshan Godaliyadda, M. P. B. Ekanayake, Vijitha HeratharXiv 2024 · DOI · arXiv

TL;DR

This paper introduces FusionNet, a novel transformer-based deep learning model for hyperspectral unmixing. FusionNet addresses the common problem of suboptimal results caused by relying on a single endmember initialization by using an ensemble of endmembers. It also incorporates spatial context from neighboring pixels in a flexible way, leading to improved abundance prediction accuracy.

What Problem Does This Paper Address?

Hyperspectral (HS) imaging captures detailed spectral information across many narrow bands, making it invaluable for applications like mineral mapping and environmental monitoring. However, each pixel in an HS image is often a mixture of different pure materials, known as endmembers. HS unmixing aims to determine the spectral signatures of these endmembers and their fractional abundances within each pixel.

A major challenge in HS unmixing is the reliance on accurate endmember signatures. Many algorithms require these signatures as input or for initialization. If the initial endmember signatures are not representative of the actual materials present or if only a single source is used for initialization, the unmixing process can lead to inaccurate abundance maps and endmember signatures. Furthermore, HS images often have low spatial resolution, making it difficult to capture local spatial relationships that can improve unmixing accuracy, especially in the presence of noise. Existing methods often use fixed neighborhood shapes (like CNN kernels), limiting their flexibility.

Methods

FusionNet is a transformer-based deep learning architecture designed to tackle these challenges. It comprises three main components:

  • Pixel Contextualizer (PC): This module injects spatial context into the unmixing process. It takes a pixel and its surrounding neighbors as input and uses a multi-head attention mechanism to generate a context-aware pixel representation. Unlike CNNs, the PC allows for flexible neighborhood configurations (e.g., doughnut, circle, random), not limited by fixed kernel shapes.
  • Abundance Predictor (AP): This component takes the context-aware pixels from the PC and predicts the abundance fractions for each endmember at a pixel level. It uses a multi-head attention block and a softmax layer to enforce Abundance Non-negativity Constraint (ANC) and Abundance Sum-to-one Constraint (ASC).
  • Signature Predictor (SP): This is a key innovation. Instead of relying on a single set of endmember signatures, the SP takes an ensemble of endmember signatures (obtained from various traditional Endmember Extraction Algorithms - EEAs) as input. It uses a multi-head attention mechanism to intelligently fuse these ensemble signatures, learning to combine them to produce more accurate endmember spectral signatures. This approach mitigates the risk of suboptimal results stemming from a single, potentially inaccurate, initial endmember set.

The entire FusionNet model is trained end-to-end. The training process is divided into two stages to ensure proper convergence:

  1. Stage 1: Focuses on guiding the Abundance Predictor by using a constrained Signature Predictor. The SP's linear layers are initially fixed, allowing it to learn to weight the ensemble components.
  2. Stage 2: Allows the Signature Predictor to fully train and refine the endmember signatures, incorporating a Minimum Volume Constraint to prevent the endmember simplex from becoming too large.

Datasets

The proposed FusionNet model was evaluated on three widely recognized real hyperspectral datasets and one synthetic dataset:

  • Samson: A dataset with three endmembers (tree, sand, water), featuring 156 bands.
  • Jasper Ridge: A dataset with four endmembers (road, soil, water, tree), with 198 bands after preprocessing.
  • Urban: A dataset with multiple ground truths, used here with 162 bands.
  • Synthetic Dataset: A dataset with 90x90 pixels, four endmembers, and 144 spectral bands, generated with specific constraints to ensure ANC and ASC.

Key Insights and Ideas

  • Ensemble of Endmembers for Robust Initialization: The most significant insight is the use of an ensemble of endmember signatures from multiple EEAs. This approach directly addresses the critical problem of initialization sensitivity in HS unmixing, leading to more robust and accurate endmember extraction.
  • Flexible Spatial Context Integration: The Pixel Contextualizer's ability to use arbitrary neighborhood shapes, rather than fixed CNN kernels, offers greater flexibility in capturing spatial information, which is crucial for improving abundance prediction, especially in complex scenes.
  • Two-Stage Training for Stability: The two-stage training strategy is crucial for stabilizing the learning process. Stage 1 guides the abundance predictor, while Stage 2 refines the endmember signatures, preventing premature convergence to suboptimal solutions.
  • Transformer Attention for Fusion: Leveraging the attention mechanism of transformers not only for spatial context but also for fusing spectral signatures from an ensemble is a powerful way to learn complex relationships and dependencies.

Key Results

FusionNet demonstrated competitive performance compared to eight state-of-the-art algorithms across the tested datasets.

  • Samson Dataset: FusionNet achieved superior performance in terms of RMSE for all endmembers, and its average RMSE was the best. While some algorithms showed slightly better SAD for specific endmembers, FusionNet's overall performance was strong.
  • Jasper Ridge Dataset: FusionNet achieved the best RMSE for tree and soil abundance, and the second-best for others, with the best average RMSE. It also showed excellent SAD performance for soil and road endmembers.
  • Urban Dataset: FusionNet achieved the best average RMSE and performed very well for grass, metal, and dirt endmembers. It also extracted all endmember signatures perfectly in terms of SAD.
  • Synthetic Dataset: FusionNet achieved the best average RMSE and the second-best RMSE for two endmembers. It also perfectly extracted all endmember signatures in terms of SAD.
  • Noise Robustness: The proposed method showed significant improvements in RMSE and comparable results in SAD under various noise levels.
  • Training Time: The paper provides detailed tables on training and prediction times, showing that FusionNet's training is more computationally intensive than some simpler methods but offers superior accuracy.

Limitations and Future Work

  • Computational Cost: While FusionNet achieves high accuracy, its transformer-based architecture and multi-stage training can be computationally more demanding than some traditional methods.
  • Hyperparameter Tuning: The performance of FusionNet is sensitive to hyperparameter settings, as indicated by the extensive hyperparameter tables provided for different datasets.
  • Future Work: The authors suggest that this work will promote further research into transformer-based endmember fusion for remote sensing applications, particularly in exploring more advanced attention mechanisms and novel ways to integrate spatial and spectral information. The flexibility in neighborhood selection for the PC also opens avenues for further exploration.

Summary updated

Abstract

In recent years, transformer-based deep learning networks have gained popularity in Hyperspectral (HS) unmixing applications due to their superior performance. The attention mechanism within transformers facilitates input-dependent weighting and enhances contextual awareness during training. Drawing inspiration from this, we propose a novel attention-based Hyperspectral Unmixing algorithm called Transformer-based Endmember Fusion with Spatial Context for Hyperspectral Unmixing (FusionNet). This network leverages an ensemble of endmembers for initial guidance, effectively addressing the issue of relying on a single initialization. This approach helps avoid suboptimal results that many algorithms encounter due to their dependence on a singular starting point. The FusionNet incorporates a Pixel Contextualizer (PC), introducing contextual awareness into abundance prediction by considering neighborhood pixels. Unlike Convolutional Neural Networks (CNNs) and traditional Transformer-based approaches, which are constrained by specific kernel or window shapes, the Fusion network offers flexibility in choosing any arbitrary configuration of the neighborhood. We conducted a comparative analysis between the FusionNet algorithm and eight state-of-the-art algorithms using three widely recognized real datasets and one synthetic dataset. The results demonstrate that FusionNet offers competitive performance compared to the other algorithms.
BibTeX
@misc{wickramathilaka2024transformer,
  author = {H. M. K. D. Wickramathilaka and Rukmal Ratnayake and D. M. U. P. Sumanasekara and Roshan Godaliyadda and M. P. B. Ekanayake and Vijitha Herath},
  title = {{Transformer based Endmember Fusion with Spatial Context for Hyperspectral Unmixing}},
  year = {2024},
  eprint = {2402.03835},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.03835},
}
Figure