Learning Multi-Modal Cross-Scale Deformable Transformer Network for Unregistered Hyperspectral Image Super-resolution

Wenqian Dong, Yang Xu, Jiahui Qu, Shaoxiong HouAAAI 2024 · DOI

TL;DR

This paper introduces M²DTN, a novel network for hyperspectral image super-resolution (HSI-SR) that addresses the challenge of unregistered multi-modal data (e.g., a low-resolution HSI and a high-resolution multispectral image). By formulating a unified model for spectral-preserving, spatially-guided registration-SR, M²DTN effectively aligns features from different modalities and leverages multi-scale transformers to reconstruct high-resolution HSI with improved spatial and spectral fidelity.

What Problem Does This Paper Address?

Hyperspectral imaging captures rich spectral information, but often suffers from low spatial resolution, limiting its use in applications like change detection or semantic segmentation. Hyperspectral Image Super-Resolution (HSI-SR) aims to enhance this spatial resolution. While fusion-based HSI-SR methods, which use auxiliary high-resolution images (like panchromatic or multispectral images), show promise, they typically assume perfect registration between the HSI and the auxiliary image. In reality, achieving such precise alignment is difficult due to varying imaging platforms, viewpoints, and atmospheric conditions. This misalignment can lead to significant performance degradation. Furthermore, many existing methods rely on Convolutional Neural Networks (CNNs), which excel at local feature extraction but struggle to capture global contextual information, further hindering HSI-SR performance.

Methods

The core of this work is the Multi-Modal Cross-Scale Deformable Transformer Network (M²DTN). It's built upon a novel Spectral-Preserving Spatial-Guided Registration-SR Unified Model (SSRU), which provides a theoretical framework for addressing unregistered HSI-SR under realistic degradation scenarios.

Key components of M²DTN include:

  • Multi-Modal Registration Deformable Module (MMRD): This module is designed to learn the spatial distribution differences between the low-resolution HSI and the high-resolution auxiliary image. It generates a deformation field to align the features of the LrHSI with the HrMSI, effectively handling misregistration.
  • Multi-Scale Feature Transformer Module (MSFT): This module utilizes a pyramid structure with Global Spectral Transformer Blocks (GSTB) to capture spatial-spectral features at multiple scales. It employs global spectral feature attention to focus on relevant spectral information and efficiently extracts multi-scale features from both modalities.
  • Cross-Scale Feature Aggregation Module (CSFA): This module aggregates the multi-scale spatial-spectral features learned by MSFT and MMRD to reconstruct the high-resolution HSI. It aims to produce more refined HSI by leveraging information across different scales.

Datasets

The authors evaluated their method on three publicly available datasets:

  • Pavia Center (PaviaC): Acquired by ROSIS sensor.
  • Houston: Acquired by ITRES-CASI HS sensor.
  • Harvard: Contains indoor and outdoor images acquired by a commercial hyperspectral camera.

For evaluation, low-resolution HSI and high-resolution MSI were generated using Wald's protocol. Elastic transformation was applied to simulate realistic distortions and misregistration.

Key Insights and Ideas

  • The critical need for handling misregistration: The paper strongly emphasizes that real-world HSI-SR applications must contend with unregistered multi-modal data, and that ignoring this can severely limit performance.
  • Unified framework for registration and SR: Formulating the problem within an SSRU framework provides a principled way to jointly address registration and super-resolution.
  • Deformable transformers for alignment: The MMRD module's use of deformable fields, guided by cross-attention principles, is a novel approach to aligning features from misregistered images.
  • Multi-scale feature fusion is key: The MSFT and CSFA modules highlight the importance of capturing and aggregating features across different spatial and spectral scales for accurate reconstruction.

Key Results

The proposed M²DTN consistently outperformed existing state-of-the-art (SOTA) HSI-SR methods on the Pavia Center, Houston, and Harvard datasets across five widely used quality assessment metrics (CC, SAM, RMSE, ERGAS, PSNR). For instance, on the PaviaC dataset, M²DTN achieved a PSNR of 45.3475, significantly higher than other methods. Visual results also demonstrated that M²DTN produced reconstructed images closest to the ground truth, with more apparent texture and fewer artifacts compared to competing methods.

Limitations and Future Work

  • Computational Cost: While transformers offer powerful feature representation, their computational complexity can be a concern, especially for hyperspectral images with many bands. The paper mentions using window self-attention and other efficiency-improving techniques, but this remains an area for further optimization.
  • Generalizability to extreme misregistration: The paper focuses on realistic degradation. The performance of M²DTN under extremely severe misregistration conditions might require further investigation.
  • Future work could explore: Extending the framework to other types of auxiliary data, investigating more efficient transformer architectures for HSI, and exploring unsupervised or self-supervised learning strategies for even more robust registration.

Summary updated

BibTeX
@inproceedings{dong2024learning,
  author = {Wenqian Dong and Yang Xu and Jiahui Qu and Shaoxiong Hou},
  title = {{Learning Multi-Modal Cross-Scale Deformable Transformer Network for Unregistered Hyperspectral Image Super-resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i2.27923},
}
Figure