THAT: Token-wise High-frequency Augmentation Transformer for Hyperspectral Pansharpening

TL;DR

This paper introduces THAT, a novel Transformer-based framework for hyperspectral pansharpening that tackles the limitations of redundant token representations and the lack of multi-scale feature modeling. By employing a Pivotal Token Selective Attention (PTSA) mechanism to focus on informative tokens and a Multi-level Variance-aware Feed-forward Network (MVFN) to enhance high-frequency details, THAT achieves state-of-the-art performance with improved spectral fidelity and spatial sharpness.

What Problem Does This Paper Address?

Hyperspectral imaging captures rich spectral information, enabling detailed material analysis. However, hyperspectral images often suffer from low spatial resolution, limiting their practical applications. Hyperspectral pansharpening aims to fuse a low-resolution hyperspectral image (LR-HSI) with a high-resolution panchromatic image (HR-PCI) to generate a high-resolution hyperspectral image (HR-HSI) with both high spatial and spectral detail. While Transformer-based models have shown promise in capturing long-range dependencies, they struggle with two key issues: (1) redundant token representations that dilute attention and (2) a lack of multi-scale feature modeling, leading to the loss of high-frequency details like material edges and textures. This paper addresses these limitations to improve the accuracy and efficiency of hyperspectral pansharpening.

Methods

The proposed framework, THAT (Token-wise High-frequency Augmentation Transformer), consists of three main stages: shallow feature extraction, deep feature extraction, and feature reconstruction. The core innovations lie in the deep feature extraction stage:

• Pivotal Token Selective Attention (PTSA): This module addresses token redundancy by dynamically prioritizing informative tokens and filtering out less relevant ones. It uses k-means clustering on attention scores to identify "pivotal" tokens that represent stronger spectral-spatial interactions. A binary mask is then applied to the attention matrix, focusing computation on these key tokens and improving efficiency and representation.
• Multi-level Variance-aware Feed-forward Network (MVFN): This module enhances high-frequency detail learning by capturing hierarchical spectral-spatial dependencies. It employs multi-scale depthwise convolutions to extract features across different receptive fields, followed by variance modeling to emphasize subtle, high-frequency variations. Pooling operations are used to suppress low-frequency components, and the aggregated features are then refined.

Datasets

The paper evaluates THAT on three publicly available hyperspectral datasets:

• Pavia Centre (PaviaC): Acquired over Pavia, Italy, with 102 spectral bands and a spatial resolution of 1.3 m.
• Pavia University (PaviaU): Also captured over Pavia, Italy, with 102 spectral bands and a spatial resolution of 1.3 m.
• Botswana: Acquired by NASA's EO-1 Hyperion sensor over the Okavango Delta, with 145 spectral bands and a spatial resolution of 30 m.

LR-HSIs and HR-PCIs are generated following Wald's protocol, involving bicubic interpolation, blurring, and downsampling for LR-HSI, and averaging visible bands for HR-PCI.

Key Insights and Ideas

• The "curse of dimensionality" in attention: Global self-attention in standard Vision Transformers can be inefficient and dilute important spectral-spatial information when applied to hyperspectral data, especially when many tokens are redundant.
• Focusing on "pivotal" tokens is key: By identifying and prioritizing tokens that exhibit strong spectral-spatial correlations (via k-means clustering on attention scores), we can significantly improve the efficiency and effectiveness of the attention mechanism.
• High-frequency details are crucial for realism: Simply capturing spectral and spatial information isn't enough; preserving and enhancing fine details like edges and textures is critical for high-quality pansharpening. MVFN's variance-aware approach directly targets this.
• Hierarchical processing matters: Processing features at multiple scales and levels of variance allows the network to better capture the complex spectral-spatial relationships inherent in hyperspectral data.

Key Results

THAT achieves state-of-the-art performance across multiple upscaling factors (×2, ×4, and ×8) on standard hyperspectral pansharpening benchmarks.

• Quantitative: The paper reports consistent improvements in PSNR, SSIM, and ERGAS metrics compared to eleven state-of-the-art methods. For example, in ablation studies, PTSA yields up to 0.88 dB PSNR improvement and ERGAS reduction, while MVFN consistently boosts performance.
• Qualitative: Visual results demonstrate THAT's ability to maintain spatial structures and spectral coherence, effectively reducing spectral distortions and enhancing spatial sharpness. Band-wise PSNR comparisons show THAT achieving the highest PSNR across most spectral bands.

Limitations and Future Work

• Computational Complexity: While THAT aims for efficiency, the paper notes that some previous methods like PLRDiff are even more computationally intensive. THAT offers a favorable trade-off with moderate complexity (1.45 M parameters and 78.42 G FLOPs).
• Future Work: The paper doesn't explicitly detail future work directions, but the success of PTSA and MVFN suggests potential for further exploration of token selection strategies and variance-aware feature enhancement in other vision tasks. The authors also highlight the need for more efficient transformer-based approaches for hyperspectral pansharpening.