Abstract
Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Owing to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers, and propose a novel backbone network called SpectralFormer. Beyond band-wise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding group-wise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layer-wise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse "soft" residuals across layers. SpectralFormer is a highly flexible backbone network, applicable to both pixel- and patch-wise inputs, achieving state-of-the-art performance across multiple benchmark datasets.
Introduction
Hyperspectral imaging (HSI) captures hundreds of narrow, contiguous wavelength bands per pixel, creating a detailed "spectral signature" that can identify materials at a highly granular level. To classify these images, researchers have traditionally relied on a variety of deep learning backbones, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs).
However, these architectures have inherent limitations in the HSI domain. CNNs are excellent at capturing spatial context but struggle to model the long-term sequential dependencies of spectral signatures. RNNs process sequences effectively but suffer from gradient vanishing and lack parallelization capabilities. While standard Vision Transformers (ViTs) solve sequence modeling and parallelization issues, they often treat spectral bands too discretely, missing the subtle, local absorption changes between adjacent bands. SpectralFormer addresses these gaps by rethinking HSI classification purely from a sequence perspective, modifying the Transformer architecture to respect the unique, continuous nature of hyperspectral data.
Methodology
SpectralFormer is a pure Transformer-based backbone that introduces two major architectural innovations designed specifically for the physical characteristics of hyperspectral data.
Group-wise Spectral Embedding (GSE)
Instead of embedding single spectral bands independently as a standard ViT would, GSE extracts features from overlapping groups of neighboring spectral bands. This allows the network to learn locally detailed absorption characteristics, preserving the physical continuity of the electromagnetic spectrum. By grouping neighboring bands, the model bridges the gap between global sequence modeling and local feature extraction.
Cross-layer Adaptive Fusion (CAF)
To prevent the network from "forgetting" crucial low-level features as it gets deeper, the authors introduce a middle-range skip connection. CAF adaptively learns to fuse "soft" residuals from shallow layers into deeper layers (e.g., skipping exactly one encoder block). This provides an optimal balance for information exchange, ensuring that memory-like components are preserved throughout the layer-wise propagation process.
Flexible Input Modes
SpectralFormer is highly versatile, supporting two primary input modes:
- Pixel-wise: Treating a single pixel's spectral signature as a 1D sequence.
- Patch-wise: Unfolding a 3D spatial-spectral patch into a sequence of 1D vectors to incorporate vital spatial context.
Experimental Results
The model was evaluated on three standard hyperspectral benchmark datasets: Indian Pines, Pavia University, and Houston2013. SpectralFormer demonstrated a massive leap over baseline Transformers and established new state-of-the-art (SOTA) performance levels:
- Indian Pines: SpectralFormer improved Overall Accuracy (OA) by roughly 10% compared to a vanilla ViT (71.86% to 81.76%).
- Pavia University: Achieved 91.07% OA, outperforming 2D-CNNs by approximately 5%.
- Houston2013: Reached 88.01% OA, significantly higher than the 83.72% achieved by 2D-CNN backbones.
Beyond raw accuracy, SpectralFormer showed remarkable stability and robustness even when training sample sizes were drastically reduced, making it highly suitable for real-world remote sensing applications where labeled data is often scarce.
Conclusion
SpectralFormer represents a paradigm shift in HSI classification by moving away from spatial-first CNNs toward a sequence-first Transformer approach that respects spectral continuity. Through Group-wise Spectral Embedding and Cross-layer Adaptive Fusion, it overcomes the limitations of standard ViTs. Future work will focus on embedding physical priors into the network to improve interpretability and exploring self-supervised learning to further reduce reliance on labeled training data.
Summary updated