HyperColorization: Propagating spatially sparse noisy spectral clues for reconstructing hyperspectral images

M. Kerem Aydin, Qi Guo, Emma Alexander

TL;DR

HyperColorization (HC) is a novel framework that reconstructs high-resolution hyperspectral images by fusing a sharp grayscale "guide" image with a small number of noisy, spatially sparse spectral samples. By performing optimization in a learned low-rank subspace and employing intelligent, image-guided sampling, the method overcomes the traditional trade-offs between spatial resolution, spectral depth, and acquisition time.

What Problem Does This Paper Address?

Hyperspectral cameras (HSCs) capture dozens or hundreds of narrow wavelength bands, providing a "spectral fingerprint" for every pixel. This is invaluable for tasks like mineralogy, medical diagnostics, and agriculture. However, HSCs face a brutal "trilemma": you can have high spatial resolution, high spectral resolution, or fast capture time, but rarely all three.

Traditional scanning HSCs (pushbroom or whiskbroom) are slow, while "snapshot" cameras often sacrifice spatial detail. Furthermore, because HSCs split incoming light into many narrow channels, each sensor pixel receives very few photons, making the data extremely susceptible to shot noise. This paper tackles the challenge of reconstructing a dense, clean hyperspectral data cube from minimal, noisy measurements by leveraging the high-resolution spatial information available in a standard grayscale camera.

Methods

The authors propose HyperColorization, which draws inspiration from classic image colorization (adding RGB to grayscale) but extends it to the high-dimensional hyperspectral domain.

  1. Optimization-based Propagation: The core assumption is that neighboring pixels with similar grayscale intensities likely share similar spectral responses. The algorithm minimizes a cost function that "spreads" the sparse spectral clues across the image, guided by the grayscale weights.
  2. Low-Rank Subspace Projection: Instead of colorizing every spectral band individually (which is slow and noisy), the authors project the data onto a low-rank basis learned via Singular Value Decomposition (SVD). This effectively "compresses" the color space, significantly speeding up computation and naturally filtering out noise components that don't align with the learned basis.
  3. Adaptive Dimensionality Estimation: A key innovation is a method to estimate the optimal number of basis vectors (rank) based on the noise level. By analyzing the "elbow" of the explained variance curve of the noisy samples, the system adaptively chooses a rank that avoids both underfitting (blurring spectra) and overfitting (preserving noise).
  4. Guided Sampling: Rather than sampling the scene uniformly, the authors use the grayscale guide to identify regions with high texture or diverse gray levels. The spectral scanner is then "guided" to sample more densely in these information-rich areas.
  5. Luminance and Edge Correction: To prevent "color bleed" across edges, they use edge-aware filtering. They also implement a renormalization step to ensure the final hyperspectral cube's integrated intensity matches the grayscale guide.

Datasets

The researchers evaluated their method using five well-known hyperspectral datasets:

  • Harvard & KAIST: Standard 31-band visible light datasets.
  • Bear & Fruit: A high-resolution 31-band dataset used for sampling optimization.
  • CAVE: Used primarily for learning the spectral basis vectors.
  • ICVL: A massive dataset with 519 bands spanning both visible and Near-Infrared (NIR) ranges, used to test the method's robustness beyond visible light.

Key Insights and Ideas

  • Noise as a Dimensionality Constraint: The authors show that the optimal reconstruction rank is inversely proportional to noise. In high-noise scenarios, a very low-rank representation (e.g., 3-5 basis vectors) actually produces a more accurate reconstruction than a high-rank one because it ignores the "dimensions" where noise dominates.
  • EMD as a Superior Metric: The paper advocates for using Earth Mover’s Distance (EMD)—also known as the Wasserstein distance—to evaluate hyperspectral quality. Unlike PSNR, which can be dominated by spatial alignment, EMD focuses on the "shape" of the spectral curve, making it a more faithful metric for spectral accuracy.
  • Quantity over Quality in Sampling: A surprising finding is that when operating under a fixed time budget, it is often better to take more sparse samples with shorter exposure times (higher noise) than fewer "clean" samples. The HC algorithm is robust enough to denoise the data, but it cannot recover spatial features that were never sampled.

Key Results

  • State-of-the-Art Performance: HC consistently outperformed existing hybrid fusion methods like SASSI and Cao et al. across PSNR, SSIM, and spectral similarity metrics.
  • Efficiency: Compared to snapshot reconstruction methods (like Choi et al.) which can take 12 hours to process a single image, HC runs in seconds, making it much more practical for real-world applications.
  • NIR Robustness: On the ICVL dataset, the method successfully propagated spectral clues into the near-infrared range (700-1100nm), despite the guide image being visible-light based.
  • Visual Fidelity: Qualitative results show that HC avoids the "color blotches" and "pixelation" artifacts common in competing methods, maintaining sharp edges and faithful color reproduction even at 1-3% sampling rates.

Limitations and Future Work

  • Hardware Requirement: The system requires a hybrid setup (two cameras or a specialized scanner), which adds complexity to the hardware design.
  • Sampling Gaps: If a unique colored object is entirely missed by the sparse sampling pattern, the algorithm has no way to "invent" its correct spectrum; it will simply assign it the spectrum of its neighbors.
  • Future Directions: The authors suggest integrating more complex spatial basis models (like deep learning priors) and expanding the work into the Short-Wave Infrared (SWIR) range, where low-rank models are less established but potentially even more impactful.

Summary updated

Abstract

Hyperspectral cameras face challenging spatial-spectral resolution trade-offs and are more affected by shot noise than RGB photos taken over the same total exposure time. Here, we present a colorization algorithm to reconstruct hyperspectral images from a grayscale guide image and spatially sparse spectral clues. We demonstrate that our algorithm generalizes to varying spectral dimensions for hyperspectral images, and show that colorizing in a low-rank space reduces compute time and the impact of shot noise. To enhance robustness, we incorporate guided sampling, edge-aware filtering, and dimensionality estimation techniques. Our method surpasses previous algorithms in various performance metrics, including SSIM, PSNR, GFC, and EMD, which we analyze as metrics for characterizing hyperspectral image quality. Collectively, these findings provide a promising avenue for overcoming the time-space-wavelength resolution trade-off by reconstructing a dense hyperspectral image from samples obtained by whisk or push broom scanners, as well as hybrid spatial-spectral computational imaging systems.
BibTeX
@misc{aydin2024hypercolorization,
  author = {M. Kerem Aydin and Qi Guo and Emma Alexander},
  title = {{HyperColorization: Propagating spatially sparse noisy spectral clues for reconstructing hyperspectral images}},
  year = {2024},
  eprint = {2403.11935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.11935},
}

Loading related content...

Figure