EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution

Xi Su, Xiangfei Shen, Mingyang Wan, Jing Nie, Lihui Chen, Haijun Liu, Xichuan ZhouAAAI 2025 · DOI

Abstract

Hyperspectral image (HSI) super-resolution (SR) is often hindered by the scarcity of large-scale training data compared to the RGB domain. This paper introduces EigenSR, a framework that utilizes "eigenimages" to bridge the gap between abundant pre-trained RGB models and data-poor HSI tasks. By decomposing HSI into spatial eigenimages and spectral bases using Singular Value Decomposition (SVD), EigenSR allows high-capacity RGB transformers to be fine-tuned on single-channel spatial structures. An Iterative Spectral Regularization (ISR) mechanism is further employed during inference to ensure spectral fidelity. Experimental results across multiple benchmarks demonstrate that EigenSR achieves state-of-the-art performance in both spatial detail and spectral accuracy.

Introduction

Hyperspectral imaging captures a wealth of spectral information, enabling applications in remote sensing, medical imaging, and material identification. However, hardware limitations often necessitate a trade-off, resulting in images with high spectral but low spatial resolution. While deep learning has significantly advanced RGB super-resolution, HSI-SR faces two primary challenges: the lack of massive HSI datasets and the structural mismatch between 3-channel RGB inputs and high-dimensional HSI cubes. This "channel gap" prevents HSI models from effectively leveraging the spatial-texture knowledge embedded in large-scale pre-trained RGB models like ImageNet-trained transformers.

The Eigenimage bridge concept for HSI-SR.

Methodology

The EigenSR framework operates in two distinct stages to transfer knowledge from the RGB domain to the HSI domain.

1. Spatial Fine-tuning via Eigenimage Bridge

The core idea is to decouple the HSI into its spatial and spectral components using Singular Value Decomposition (SVD). The HSI is represented as a set of eigenimages and a corresponding spectral basis. The authors prove that these eigenimages preserve the essential spatial structures and degradation patterns of the original HSI. A pre-trained Image Processing Transformer (IPT), originally trained on millions of RGB images, is then fine-tuned on these single-channel eigenimages. This allows the model to adapt its "spatial intelligence" to HSI data without requiring a large-scale HSI dataset.

2. Iterative Spectral Regularization (ISR)

Super-resolving eigenimages independently can lead to spectral distortion. To mitigate this, EigenSR introduces an iterative inference process. The model super-resolves the top $R$ eigenimages, which are then combined with the low-resolution input to update the spectral basis. This loop, typically repeated five times, progressively refines spectral fidelity while maintaining sharp spatial details. The framework offers two variants: EigenSR-\alpha (a fast, one-pass version) and EigenSR-\beta (the iterative version for maximum accuracy).

The two-stage EigenSR framework: Fine-tuning and Iterative Spectral Regularization.

Experiments and Results

EigenSR was evaluated on diverse benchmarks, including natural scenes (ARAD_1K, CAVE, Harvard) and remote sensing datasets (Pavia, DC Mall, Chikusei). For remote sensing tasks, the model was fine-tuned using the RESISC45 dataset.

Key results include: * State-of-the-Art Performance: EigenSR-\beta consistently outperformed existing HSI-SR methods such as ESSAformer and TSBSR. On the ARAD_1K dataset ($\times4$ scale), it achieved a PSNR of 40.46 dB, surpassing the next best supervised method. * Spectral Fidelity: Spectral Angle Mapper (SAM) scores indicated that EigenSR preserves "color" accuracy better than competitors, largely due to the ISR stage. * Generalization: Models fine-tuned on one dataset (e.g., ARAD_1K) showed robust performance on others (e.g., CAVE) without additional training, demonstrating the effectiveness of the eigenimage-based transfer learning.

Conclusion

EigenSR effectively addresses the data scarcity and structural mismatch problems in HSI-SR by using eigenimages as a bridge to pre-trained RGB models. By operating in a spectral-agnostic spatial domain, the method leverages high-level spatial priors while maintaining physical spectral constraints. Future work may focus on optimizing inference speed for real-time applications and automating the selection of the optimal rank $R$ for different scenes.

Summary updated

BibTeX
@inproceedings{su2025eigensr,
  author = {Xi Su and Xiangfei Shen and Mingyang Wan and Jing Nie and Lihui Chen and Haijun Liu and Xichuan Zhou},
  title = {{EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32755},
}
Figure