SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers

Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot

TL;DR

SpectralFormer is a novel Transformer-based backbone designed specifically to handle the sequential nature of hyperspectral (HS) image data. By introducing group-wise spectral embeddings and a cross-layer adaptive fusion mechanism, it outperforms traditional CNNs and standard Vision Transformers (ViT) in identifying materials with highly similar spectral signatures.

What Problem Does This Paper Address?

Hyperspectral imaging captures hundreds of narrow, contiguous spectral bands for every pixel, creating a "spectral signature" that acts like a fingerprint for materials. Traditionally, researchers have used Convolutional Neural Networks (CNNs) for classification because they excel at capturing spatial context. However, CNNs are not inherently designed to model the sequential dependencies between spectral bands.

While Recurrent Neural Networks (RNNs) can handle sequences, they are difficult to train in parallel and often struggle with "forgetting" information over long sequences. Recently, Transformers have dominated sequence modeling, but applying them directly to HS data (like a standard Vision Transformer) often fails to capture local spectral fluctuations and loses valuable information as the network gets deeper.

Methods

The authors propose SpectralFormer, which modifies the standard Transformer architecture with two key innovations:

  1. Group-wise Spectral Embedding (GSE): Standard Transformers typically treat each band (or patch) as an independent token. GSE instead groups neighboring bands together before embedding them. This allows the model to capture "spectrally local" information—essentially the local slope and shape of the spectral curve—which is critical for distinguishing between similar materials.
  2. Cross-layer Adaptive Fusion (CAF): To prevent the loss of information in deeper layers, the authors designed a "middle-range" skip connection. Unlike standard residuals that stay within a single block, CAF adaptively fuses features from shallow layers (e.g., Encoder 1) into deeper layers (e.g., Encoder 3). This acts as a "memory" mechanism, ensuring that fine-grained spectral details aren't smoothed away during processing.

SpectralFormer is highly flexible; it can process data as a single 1D pixel (spectral-only) or as a 3D patch (spatial-spectral) by unfolding the spatial dimensions into the sequence.

Datasets

The model was evaluated on three benchmark hyperspectral datasets:

  • Indian Pines: 200 spectral bands; primarily agricultural land cover.
  • Pavia University: 103 spectral bands; an urban scene captured over Italy.
  • Houston 2013: 144 spectral bands; a complex urban/rural dataset provided by the IEEE GRSS Data Fusion Contest.

Key Insights and Ideas

  • Spectral as Sequence: The core "aha moment" is treating the spectral dimension not just as a stack of images, but as a sequence where the order and relationship between bands carry physical meaning (absorption characteristics).
  • Local vs. Global: While the Transformer’s self-attention mechanism is great at finding global relationships, the GSE module proves that local spectral context is equally vital for hyperspectral data.
  • Information Transitivity: The authors found that standard Transformers suffer from a "semantic gap" between shallow and deep features. The CAF module bridges this gap, allowing the model to converge faster and achieve higher accuracy with fewer training epochs.

Key Results

SpectralFormer significantly outperformed both traditional classifiers (SVM, KNN) and modern deep learning backbones (1D/2D CNNs, RNNs, and GCNs).

  • Accuracy Boost: On the Indian Pines dataset, SpectralFormer achieved an Overall Accuracy (OA) of 81.76%, a massive jump compared to the standard Transformer's 71.86%.
  • State-of-the-Art: It consistently beat the previous state-of-the-art (miniGCN and 2D-CNNs) across all three datasets, often by 2-5% in overall accuracy.
  • Visual Quality: Qualitative classification maps showed that SpectralFormer produces much cleaner results with fewer "salt-and-pepper" noise errors compared to other methods, particularly in complex urban areas.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Computational Complexity: The self-attention mechanism has a quadratic complexity relative to the number of spectral bands, which can be demanding for very high-resolution data.
  • Hyperparameter Sensitivity: The performance of the CAF module depends on the number of layers skipped, which currently requires manual tuning.
  • Future Directions: The team plans to explore lightweight versions of the architecture and incorporate self-supervised learning to reduce the reliance on large amounts of labeled training data. They also aim to integrate more "physical" prior knowledge of spectral bands directly into the network layers.

Summary updated

Abstract

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Due to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers and propose a novel backbone network called SpectralFormer . Beyond bandwise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding groupwise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layerwise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse “soft” residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixelwise and patchwise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.
BibTeX
@article{hong2022spectralformer,
  author = {Danfeng Hong and Zhu Han and Jing Yao and Lianru Gao and Bing Zhang and Antonio Plaza and Jocelyn Chanussot},
  title = {{SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2022},
  url = {https://openreview.net/forum?id=G96PXlomyC},
}

Loading related content...

Figure