HyperColorization: Propagating spatially sparse noisy spectral clues for reconstructing hyperspectral images

M. Kerem Aydin, Qi Guo, Emma Alexander

Abstract

Hyperspectral cameras face challenging spatial-spectral resolution trade-offs and are more affected by shot noise than RGB photos taken over the same total exposure time. Here, we present a colorization algorithm to reconstruct hyperspectral images from a grayscale guide image and spatially sparse spectral clues. We demonstrate that our algorithm generalizes to varying spectral dimensions for hyperspectral images, and show that colorizing in a low-rank space reduces compute time and the impact of shot noise. To enhance robustness, we incorporate guided sampling, edge-aware filtering, and dimensionality estimation techniques. Our method surpasses previous algorithms in various performance metrics, including SSIM, PSNR, GFC, and EMD, which we analyze as metrics for characterizing hyperspectral image quality. Collectively, these findings provide a promising avenue for overcoming the time-space-wavelength resolution trade-off by reconstructing a dense hyperspectral image from samples obtained by whisk or push broom scanners, as well as hybrid spatial-spectral computational imaging systems.

Introduction

Hyperspectral cameras (HSCs) are essential for capturing "spectral fingerprints" in fields like medical diagnostics and mineralogy. However, they face a fundamental "trilemma": balancing spatial resolution, spectral depth, and acquisition speed. Traditional scanning HSCs are slow, while snapshot cameras often sacrifice spatial detail. Furthermore, because HSCs split light into many narrow channels, each pixel receives few photons, making the data highly susceptible to shot noise.

HyperColorization (HC) is a novel framework designed to overcome these trade-offs. It reconstructs high-resolution hyperspectral images by fusing a sharp grayscale "guide" image with a small number of noisy, spatially sparse spectral samples. By performing optimization in a learned low-rank subspace and employing intelligent, image-guided sampling, HC achieves dense, clean hyperspectral data cubes from minimal measurements.

The HyperColorization Pipeline

The core of the HC method is an optimization-based propagation that assumes neighboring pixels with similar grayscale intensities likely share similar spectral responses. This process is enhanced by several key technical innovations:

  1. Low-Rank Subspace Projection: Instead of processing every spectral band individually, the data is projected onto a low-rank basis learned via Singular Value Decomposition (SVD). This filters out noise and accelerates computation.
  2. Adaptive Dimensionality Estimation: The system adaptively estimates the optimal number of basis vectors (rank) based on noise levels, avoiding both underfitting and overfitting.
  3. Guided Sampling: The grayscale guide identifies information-rich regions (high texture or diverse gray levels) to guide the spectral scanner to sample more densely where it matters most.
Overview of the HyperColorization pipeline, illustrating the integration of hardware components and software algorithms for spectral reconstruction.

Key Insights and Technical Validation

A fundamental assumption of the HC framework is the correlation between spatial intensity differences in the guide image and spectral differences. Validation across various bands shows high correlation, particularly in the visible spectrum (0.99), which supports the use of grayscale images to guide spectral propagation.

Correlation analysis between normalized absolute monochrome difference and spectral difference across visible and NIR bands, validating the core propagation assumption.

Additional insights from the research include:

  • Noise as a Constraint: In high-noise scenarios, a lower-rank representation (3-5 basis vectors) often produces more accurate reconstructions by ignoring noise-dominated dimensions.
  • Quantity over Quality: When capture time is limited, it is often better to take more sparse, noisy samples than fewer clean ones. The HC algorithm is robust enough to denoise the data, but cannot recover spatial features that were never sampled.
  • Metric Superiority: The authors advocate for Earth Mover’s Distance (EMD) over PSNR for evaluating hyperspectral quality, as EMD better captures the accuracy of the spectral curve shape.

Results and Performance

HyperColorization consistently outperforms existing hybrid fusion methods like SASSI across PSNR, SSIM, and spectral similarity metrics. Notably, while some snapshot reconstruction methods can take up to 12 hours per image, HC runs in seconds. The method also demonstrates robustness in the near-infrared range (700-1100nm) and maintains high visual fidelity—avoiding common artifacts like pixelation or color blotches—even at sampling rates as low as 1-3%.

Conclusion

HyperColorization provides a practical path for high-resolution hyperspectral imaging by leveraging the strengths of hybrid hardware systems. While it currently requires specialized setups and can be limited if unique spectral features are entirely missed during sampling, future work aims to integrate deep learning priors and expand capabilities into the Short-Wave Infrared (SWIR) range.

Summary updated

Abstract

Hyperspectral cameras face challenging spatial-spectral resolution trade-offs and are more affected by shot noise than RGB photos taken over the same total exposure time. Here, we present a colorization algorithm to reconstruct hyperspectral images from a grayscale guide image and spatially sparse spectral clues. We demonstrate that our algorithm generalizes to varying spectral dimensions for hyperspectral images, and show that colorizing in a low-rank space reduces compute time and the impact of shot noise. To enhance robustness, we incorporate guided sampling, edge-aware filtering, and dimensionality estimation techniques. Our method surpasses previous algorithms in various performance metrics, including SSIM, PSNR, GFC, and EMD, which we analyze as metrics for characterizing hyperspectral image quality. Collectively, these findings provide a promising avenue for overcoming the time-space-wavelength resolution trade-off by reconstructing a dense hyperspectral image from samples obtained by whisk or push broom scanners, as well as hybrid spatial-spectral computational imaging systems.
BibTeX
@misc{aydin2024hypercolorization,
  author = {M. Kerem Aydin and Qi Guo and Emma Alexander},
  title = {{HyperColorization: Propagating spatially sparse noisy spectral clues for reconstructing hyperspectral images}},
  year = {2024},
  eprint = {2403.11935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.11935},
}
Figure