HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening

Wele Gedara Chaminda Bandara, Vishal M. Patel

Abstract

Pansharpening aims to fuse a registered high-resolution panchromatic image (PAN) with a low-resolution hyper-spectral image (LR-HSI) to generate an enhanced HSI with high spectral and spatial resolution. Existing pansharpening approaches neglect using an attention mechanism to transfer HR texture features from PAN to LR-HSI features, resulting in spatial and spectral distortions. In this paper, we present a novel attention mechanism for pansharpening called HyperTransformer, in which features of LR-HSI and PAN are formulated as queries and keys in a transformer, respectively. HyperTransformer consists of three main modules, namely two separate feature extractors for PAN and HSI, a multi-head feature soft-attention module, and a spatial-spectral feature fusion module. Such a network improves both spatial and spectral quality measures of the pansharpened HSI by learning cross-feature space dependencies and long-range details of PAN and LR-HSI. Furthermore, HyperTransformer can be utilized across multiple spatial scales at the backbone for obtaining improved performance. Extensive experiments conducted on three widely used datasets demonstrate that HyperTransformer achieves significant improvement over the state-of-the-art methods on both spatial and spectral quality measures.

Introduction

In remote sensing, a fundamental trade-off exists between spatial and spectral resolution. Panchromatic (PAN) images offer high spatial resolution but lack spectral depth, while Hyperspectral Images (HSI) provide rich spectral information at the cost of spatial detail. Pansharpening is the process of fusing these two sources to create a high-resolution hyperspectral product.

Traditional convolutional neural networks (CNNs) often struggle with this task due to their limited receptive fields, which hinder their ability to capture long-range dependencies between sharp PAN textures and broad HSI spectral patterns. This often results in blurry edges or spectral shifts. The HyperTransformer addresses these limitations by introducing an attention-based mechanism designed specifically for feature transfer.

Comparison of traditional fusion methods (a, b) with the proposed HyperTransformer architecture (c).

Methods

The HyperTransformer architecture departs from simple image concatenation, instead utilizing a Transformer-based attention mechanism to precisely transfer high-resolution (HR) textures.

Query, Key, and Value Formulation

The model redefines the standard Transformer components for the pansharpening task:

  • Queries (Q): Derived from the low-resolution HSI features, representing the spectral base to be enhanced.
  • Keys (K): Derived from a degraded (down-sampled then up-sampled) version of the PAN image.
  • Values (V): Derived from the original HR PAN image, serving as the source of sharp textural details.

Multi-Head Feature Soft-Attention (MHFSA)

The MHFSA module calculates cross-correlation between HSI and PAN features. By identifying which HR textures from the PAN image are most relevant to specific spectral regions in the HSI, the model ensures that texture transfer is spectrally consistent and avoids local "smearing."

Multi-Scale Fusion and Loss Functions

To capture both fine-grained and structural details, HyperTransformers are integrated at multiple spatial scales (typically x1, x2, and x4) within a backbone network. The training process is guided by a combination of $L_1$ reconstruction loss, VGG perceptual loss for visual realism, and a novel Transfer Perceptual Loss to ensure texture fidelity.

Experimental Results

The model was evaluated on three benchmark datasets: Pavia Center, Botswana, and Chikusei. Quantitative and qualitative results demonstrate that HyperTransformer consistently outperforms state-of-the-art (SOTA) methods.

Qualitative and quantitative comparison of HyperTransformer against SOTA methods across different datasets.

Quantitative Performance

On the Pavia Center dataset, HyperTransformer improved the Spectral Angle Mapper (SAM) score—a critical metric for spectral purity—by approximately 26.9% over previous SOTA models. Significant gains were also observed in Peak Signal-to-Noise Ratio (PSNR) and Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS).

Qualitative Analysis

Mean Absolute Error (MAE) maps reveal that HyperTransformer produces significantly fewer reconstruction errors compared to models like PanNet or DARN. The resulting images exhibit superior sharpness and maintain color accuracy relative to the ground truth.

Conclusion

HyperTransformer demonstrates that attention mechanisms are highly effective for textural feature transfer in pansharpening. By leveraging long-range dependencies and multi-scale fusion, the model achieves a new state-of-the-art in both spatial and spectral quality. Implementation code and pretrained weights are available at https://github.com/wgcban/HyperTransformer.

One identified limitation is increased error in the Ultraviolet (UV) and Infrared (IR) regions, likely due to the limited spectral sensitivity of standard PAN sensors. Future work may explore the use of multi-band panchromatic data or specialized attention masks to better address these spectral bands.

Summary updated

Abstract

Pansharpening aims to fuse a registered high-resolution panchromatic image (PAN) with a low-resolution hyper-spectral image (LR-HSI) to generate an enhanced HSI with high spectral and spatial resolution. Existing pansharpening approaches neglect using an attention mechanism to transfer HR texture features from PAN to LR-HSI features, resulting in spatial and spectral distortions. In this paper, we present a novel attention mechanism for pansharpening called HyperTransformer, in which features of LR-HSI and PAN are formulated as queries and keys in a transformer, respectively. HyperTransformer consists of three main modules, namely two separate feature extractors for PAN and HSI, a multi-head feature soft-attention module, and a spatial-spectral feature fusion module. Such a network improves both spatial and spectral quality measures of the pansharpened HSI by learning cross-feature space dependencies and long-range details of PAN and LR-HSI. Furthermore, HyperTransformer can be utilized across multiple spatial scales at the backbone for obtaining improved performance. Extensive experiments conducted on three widely used datasets demonstrate that HyperTransformer achieves significant improvement over the state-of-the-art methods on both spatial and spectral quality measures. Implementation code and pretrained weights can be accessed at https://github.com/wgcban/HyperTransformer.
BibTeX
@inproceedings{bandara2022hypertransformer,
  author = {Wele Gedara Chaminda Bandara and Vishal M. Patel},
  title = {{HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=hJP646P6rJRl},
}
Figure