Bidirectional Dilation Transformer for Multispectral and Hyperspectral Image Fusion

Shangqi Deng, Liang-Jian Deng, Xiao Wu, Ran Ran, Rui Wen

Abstract

This paper introduces the Bidirectional Dilation Transformer (BDT), a novel architecture designed to solve the Multispectral and Hyperspectral Image Fusion (MHIF) problem. By proposing Dilation Spatial Multi-head Self-Attention (D-Spa) and Grouped Spectral Multi-head Self-Attention (G-Spe), the model effectively captures long-range spatial dependencies and localized spectral characteristics. BDT achieves state-of-the-art (SOTA) performance on standard benchmarks while maintaining a highly efficient parameter count.

Introduction

In fields such as remote sensing and biomedical imaging, hyperspectral images (HSIs) are essential for their dense spectral information. However, physical sensor constraints create a trade-off between spatial and spectral resolution. Multispectral and Hyperspectral Image Fusion (MHIF) aims to computationally fuse a low-resolution HSI (LR-HSI) with a high-resolution MSI (HR-MSI) to generate a high-resolution HSI (HR-HSI). While Convolutional Neural Networks (CNNs) and standard Vision Transformers have been applied to this task, they often struggle to balance global context with fine-grained local details.

Proposed Method: Custom Attention Modules

The core innovation of the BDT architecture lies in two custom Transformer modules designed to address the limitations of standard spatial and spectral attention.

Comparison of standard attention mechanisms (a, c) with the proposed D-Spa (b) and G-Spe (d) modules.

Dilation Spatial Multi-head Self-Attention (D-Spa)

Standard spatial self-attention (Spa-MSA) is typically restricted to small local windows to maintain computational efficiency, which limits its receptive field. D-Spa employs a "hollow" strategy inspired by dilated convolutions. By using a dilation rate (e.g., $d=2$), the module expands its receptive field to capture longer-range spatial dependencies without increasing the number of parameters or the computational cost. This allows the model to act as a content-aware, dynamic convolution.

Grouped Spectral Multi-head Self-Attention (G-Spe)

Standard spectral attention (Spe-MSA) calculates global channel-wise relationships, which can wash out local spatial details. G-Spe instead subdivides the spatial dimensions into regular groups and performs spectral attention within those groups. This forces the network to learn localized spectral behaviors and extract latent features that global attention would otherwise miss.

Bimodal Feature Fusion (BFF)

The BDT architecture utilizes a hierarchical two-stage process: Bimodal Feature Extraction (BFE) and Bimodal Feature Fusion (BFF). The BFF stage is critical for integrating multi-scale features extracted from the MSI and HSI branches.

The Bimodal Feature Fusion (BFF) stage, illustrating the multi-scale fusion of features to reconstruct the final high-resolution HSI.

As shown in the fusion architecture, features from different levels ($\mathcal{D}_i$ and $\mathcal{G}_i$) are concatenated and fused using multi-scale kernels ($3 \times 3$ and $5 \times 5$), followed by upsampling ($U_{\uparrow 2}$) to match the target resolution. This multi-scale approach ensures that both coarse structural information and fine spectral details are preserved in the final output $\tilde{\mathcal{X}}$.

Experimental Results

The BDT model was evaluated on the CAVE and Harvard datasets using a standard simulation protocol involving Gaussian blurring and downsampling.

  • CAVE Dataset: BDT achieved a peak signal-to-noise ratio (PSNR) of 52.30 dB, significantly outperforming the next best model, MoG-DCN (51.63 dB).
  • Harvard Dataset: The model reached a PSNR of 48.83 dB, surpassing HSRNet (48.29 dB).
  • Efficiency: BDT is highly efficient, utilizing only 2.668M parameters. For comparison, it outperforms models like DHIF, which has nearly ten times the parameter count (22.46M).

Conclusion

The Bidirectional Dilation Transformer (BDT) successfully addresses the spatial-spectral trade-off in MHIF by leveraging dilated spatial attention and grouped spectral attention. Its modular design allows D-Spa and G-Spe to be used as plug-and-play components in other Transformer-based architectures. Future work will focus on adapting BDT for real-world scenarios involving sensor noise and spatial misalignments between MSI and HSI sensors.

Summary updated

Abstract

Transformer-based methods have proven to be ef-1 fective in achieving long-distance modeling, cap-2 turing the spatial and spectral information, and 3 exhibiting strong inductive bias in various com-4 puter vision tasks. Generally, the Transformer 5 model includes two common modes of multi-head 6 self-attention (MSA): spatial MSA (Spa-MSA) and 7 spectral MSA (Spe-MSA). However, Spa-MSA 8 is computationally efficient but limits the global 9 spatial response within a local window. On the 10 other hand, Spe-MSA can calculate channel self-11 attention to accommodate high-resolution images, 12 but it disregards the crucial local information that 13 is essential for low-level vision tasks. In this study, 14 we propose a bidirectional dilation Transformer 15 (BDT) for multispectral and hyperspectral image 16 fusion (MHIF), which aims to leverage the advan-17 tages of both MSA and the latent multiscale infor-18 mation specific to MHIF tasks. The BDT consists 19 of two designed modules: the dilation Spa-MSA 20 (D-Spa), which dynamically expands the spatial re-21 ceptive field through a given hollow strategy, and 22 the grouped Spe-MSA (G-Spe), which extracts la-23 tent features within the feature map and learns lo-24 cal data behavior. Additionally, to fully exploit 25 the multiscale information from both inputs with 26 different spatial resolutions, we employ a bidirec-27 tional hierarchy strategy in the BDT, resulting in 28 improved performance. Finally, extensive experi-29 ments on two commonly used datasets, CAVE and 30 Harvard, demonstrate the superiority of BDT both 31 visually and quantitatively.
BibTeX
@inproceedings{deng2023bidirectional,
  author = {Shangqi Deng and Liang-Jian Deng and Xiao Wu and Ran Ran and Rui Wen},
  title = {{Bidirectional Dilation Transformer for Multispectral and Hyperspectral Image Fusion}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/404},
}
Figure