NukesFormers: Unpaired Hyperspectral Image Generation With Nonuniform Domain Alignment

Jiaojiao Li, Shiyao Duan, Haitao Xu, Rui Song, Yunsong Li, Qian DuIEEE TGRS 2025 · DOI

TL;DR

This paper introduces NukesFormers, a novel framework for generating high-quality hyperspectral images (HSIs) from unpaired RGB images. It tackles the challenge of domain misalignment by decomposing the problem into range-space interactions and null-space compensations, using a combination of contrastive learning and a new Non-uniform Kolmogorov-Arnold Network (Nukes) architecture.

What Problem Does This Paper Address?

Acquiring perfectly aligned pairs of RGB and hyperspectral images (HSIs) is a significant bottleneck for training current hyperspectral image generation (HIG) models. This difficulty, coupled with the inherent complexity of aligning features across these domains, has hindered progress in unpaired HIG (UnHIG). Existing methods struggle to effectively capture and leverage subtle, high-frequency spectral information that is often degraded or lost during the generation process, leading to suboptimal reconstruction quality.

Methods

The core of NukesFormers lies in its innovative approach to handling unpaired data:

  • Range-Null Space Decomposition (RND): The problem is reframed by decomposing the mapping between RGB and HSI into "range-space" interactions (capturing consistent features) and "null-space" compensations (addressing degraded and high-frequency components).
  • Dual-Dimensional Contrastive Prior Module (DCPM): This module establishes robust constraints by learning mutual information within both RGB and HSI domains of a single scene, while also modeling relationships between representations across different scenes. This helps align geometric and spectral distributions.
  • Gabor Kernel-based Multi-head Self-attention (G-MSA): This component adaptively focuses on relevant frequency intervals for specific objects by using Gabor kernels, effectively separating and leveraging high-frequency spectral information.
  • Non-uniform Kolmogorov-Arnold Networks (Nukes): This novel architecture, built upon Kolmogorov-Arnold Networks (KANs), is designed to dynamically adjust control points. This allows it to better capture and compensate for the complex, high-dimensional features within the null space, which are crucial for accurate spectral reconstruction.

Datasets

The proposed NukesFormers framework was evaluated on the following datasets:

  • NTIRE 2020 Clean Track
  • NTIRE 2022
  • CAVE

Key Insights and Ideas

  • RND as a Unifying Framework: Decomposing the HIG problem into range and null spaces provides a principled way to handle unpaired data, separating the well-behaved components from the challenging ones.
  • The Power of Non-uniformity: Traditional KANs are powerful but can be rigid. Introducing non-uniformity in the control points of Nukes allows for much more adaptive and precise fitting of complex spectral features.
  • Dual-Domain Contrastive Learning is Key: Simply aligning features within a single scene isn't enough. Learning across different scenes with DCPM provides a more generalized and robust constraint for unpaired data.
  • Gabor Kernels for Targeted Frequency Analysis: Instead of relying on broad frequency transformations, Gabor kernels offer a more localized and object-aware approach to extracting critical high-frequency spectral information.

Key Results

NukesFormers achieved state-of-the-art performance on the evaluated datasets. Specifically, on the NTIRE 2020 Clean Track, it demonstrated:

  • A reduction of 0.0050 in RMSE.
  • A reduction of 0.0047 in MRAE.
  • An improvement of 1.31 in PSNR.
  • An improvement of 0.0172 in SSIM.

The paper also highlights that while NukesFormers might have a slightly higher parameter count during training compared to some methods, its inference efficiency is significantly improved, especially when adversarial loss is not used during testing.

Limitations and Future Work

  • Parameter Count: While the paper argues for efficiency, the training parameter count is noted to be potentially higher than some leading methods. However, the inference count is significantly reduced.
  • Future Work: The paper doesn't explicitly detail future work, but the success of the RND framework and the novel Nukes architecture suggests potential for application in other domain adaptation and image generation tasks. Further exploration of the dynamic control point generation in Nukes could also be a fruitful avenue.

Summary updated

Abstract

The persistent challenge of acquiring precisely coregistered RGB-hyperspectral image (HSI) pairs has significantly impeded the practical deployment of current data-driven Hyperspectral Image Generation (HIG) networks in engineering applications. Gleichzeitig, the ill-posed nature of the aligning constraints, compounded with the complexities of mining cross-domain features, also hinders the advancement of unpaired HIG (UnHIG) tasks. In this paper, we conquer these challenges by reformulating the UnHIG through Range-Null Space Decomposition (RND), modeling range-space feature interaction and null-space compensations. Specifically, the introduced contrastive learning effectively aligns the geometric and spectral distributions of unpaired data by building the interaction of range space, considering the consistent feature in degradation process. Our Dual-Dimensional Contrastive Prior Module (DCPM) captures mutual information within RGB and HSI domains of a single scene while modeling relationships between internal representations across different scenes, thereby constructing comprehensive cross-domain constraints. Furthermore, the Gabor kernel-based multi-head self-attention (G-MSA) adaptively separates high-frequency components, guiding subsequent modules to concentrate on relevant frequency intervals for target objects. Then, we propose a novel Non-uniform Kolmogorov-Arnold Networks (Nukes) to exhaustively excavate null-space components(degraded/high-frequency representations) through dual-domain frequency mapping. The proposed method was evaluated on three established datasets: NTIRE 2020 ’Clean’ track, NTIRE 2022, CAVE and Grss_dfc_2018. To assess real-world applicability, ratio experiments were conducted by changing the proportion of RGB and HSI. These experiments demonstrate that our approach achieves state-of-the-art performance in UnHIG.
BibTeX
@article{li2025nukesformers,
  author = {Jiaojiao Li and Shiyao Duan and Haitao Xu and Rui Song and Yunsong Li and Qian Du},
  title = {{NukesFormers: Unpaired Hyperspectral Image Generation With Nonuniform Domain Alignment}},
  journal = {IEEE Transactions on Geoscience and Remote Sensing},
  year = {2025},
  doi = {10.1109/tgrs.2025.3634312},
}
Figure