Abstract
This paper introduces the Bidirectional Dilation Transformer (BDT), a novel architecture designed to solve the Multispectral and Hyperspectral Image Fusion (MHIF) problem. By proposing Dilation Spatial Multi-head Self-Attention (D-Spa) and Grouped Spectral Multi-head Self-Attention (G-Spe), the model effectively captures long-range spatial dependencies and localized spectral characteristics. BDT achieves state-of-the-art (SOTA) performance on standard benchmarks while maintaining a highly efficient parameter count.
Introduction
In fields such as remote sensing and biomedical imaging, hyperspectral images (HSIs) are essential for their dense spectral information. However, physical sensor constraints create a trade-off between spatial and spectral resolution. Multispectral and Hyperspectral Image Fusion (MHIF) aims to computationally fuse a low-resolution HSI (LR-HSI) with a high-resolution MSI (HR-MSI) to generate a high-resolution HSI (HR-HSI). While Convolutional Neural Networks (CNNs) and standard Vision Transformers have been applied to this task, they often struggle to balance global context with fine-grained local details.
Proposed Method: Custom Attention Modules
The core innovation of the BDT architecture lies in two custom Transformer modules designed to address the limitations of standard spatial and spectral attention.
Dilation Spatial Multi-head Self-Attention (D-Spa)
Standard spatial self-attention (Spa-MSA) is typically restricted to small local windows to maintain computational efficiency, which limits its receptive field. D-Spa employs a "hollow" strategy inspired by dilated convolutions. By using a dilation rate (e.g., $d=2$), the module expands its receptive field to capture longer-range spatial dependencies without increasing the number of parameters or the computational cost. This allows the model to act as a content-aware, dynamic convolution.
Grouped Spectral Multi-head Self-Attention (G-Spe)
Standard spectral attention (Spe-MSA) calculates global channel-wise relationships, which can wash out local spatial details. G-Spe instead subdivides the spatial dimensions into regular groups and performs spectral attention within those groups. This forces the network to learn localized spectral behaviors and extract latent features that global attention would otherwise miss.
Bimodal Feature Fusion (BFF)
The BDT architecture utilizes a hierarchical two-stage process: Bimodal Feature Extraction (BFE) and Bimodal Feature Fusion (BFF). The BFF stage is critical for integrating multi-scale features extracted from the MSI and HSI branches.
As shown in the fusion architecture, features from different levels ($\mathcal{D}_i$ and $\mathcal{G}_i$) are concatenated and fused using multi-scale kernels ($3 \times 3$ and $5 \times 5$), followed by upsampling ($U_{\uparrow 2}$) to match the target resolution. This multi-scale approach ensures that both coarse structural information and fine spectral details are preserved in the final output $\tilde{\mathcal{X}}$.
Experimental Results
The BDT model was evaluated on the CAVE and Harvard datasets using a standard simulation protocol involving Gaussian blurring and downsampling.
- CAVE Dataset: BDT achieved a peak signal-to-noise ratio (PSNR) of 52.30 dB, significantly outperforming the next best model, MoG-DCN (51.63 dB).
- Harvard Dataset: The model reached a PSNR of 48.83 dB, surpassing HSRNet (48.29 dB).
- Efficiency: BDT is highly efficient, utilizing only 2.668M parameters. For comparison, it outperforms models like DHIF, which has nearly ten times the parameter count (22.46M).
Conclusion
The Bidirectional Dilation Transformer (BDT) successfully addresses the spatial-spectral trade-off in MHIF by leveraging dilated spatial attention and grouped spectral attention. Its modular design allows D-Spa and G-Spe to be used as plug-and-play components in other Transformer-based architectures. Future work will focus on adapting BDT for real-world scenarios involving sensor noise and spatial misalignments between MSI and HSI sensors.
Summary updated