l-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement

Xin Miao, Xin Yuan, Yunchen Pu, Vassilis AthitsosICCV 2019

TL;DR

This paper introduces λ-net, a novel deep learning architecture for reconstructing hyperspectral images from a single snapshot measurement. λ-net significantly outperforms existing methods in both image quality and reconstruction speed, enabling real-time hyperspectral imaging.

What Problem Does This Paper Address?

Hyperspectral imaging captures detailed spectral information across a scene, but traditional methods are slow and data-intensive. Snapshot compressive-spectral imaging (SCI) offers a solution by capturing a 3D hyperspectral signal (spatial x, y and spectral λ) with a single 2D snapshot. However, reconstructing the full hyperspectral data-cube from this compressed measurement has been a bottleneck due to slow and often low-quality reconstruction algorithms. This paper addresses the challenge of achieving both high-quality and high-speed hyperspectral image reconstruction from SCI measurements.

Methods

The proposed solution, λ-net, is a dual-stage generative model:

  • Reconstruction Stage: This stage employs a conditional Generative Adversarial Network (cGAN) with a U-net architecture as the generator.
  • Self-Attention GAN: To capture both spatial and spectral correlations, a self-attention mechanism is integrated into the U-net. This allows the network to learn long-range dependencies within the spectral images.
  • Hierarchical Channel Reconstruction (HCR): Instead of reconstructing all spectral channels at once, HCR progressively reconstructs them in stages (e.g., 1→6→12→24 channels). This decomposes the complex problem into a series of more manageable sub-problems, improving accuracy.
  • Conditional GAN: The cGAN framework allows the generator to produce hyperspectral images conditioned on the input measurement and masks, with the discriminator helping to ensure realistic outputs.
  • Refinement Stage: A shallower U-net with residual learning is used to further enhance the quality of the reconstructed hyperspectral images. This stage processes each spectral channel independently, allowing for efficient refinement.

Datasets

  • Training and Validation: The authors used a dataset from [5], which consists of hyperspectral images. To unify spectral wavelengths across different imaging systems, spectral interpolation was applied. Data augmentation techniques like random flipping, rotation, scaling, and translation were also employed. The augmented data has a size of 1392 × 1300 × 24.
  • Real Data: The authors also used real data captured by a CASSI camera for evaluation.

Key Insights and Ideas

  • Deep Learning for SCI: The paper demonstrates the significant potential of deep learning, particularly GANs and U-nets, for tackling the challenging SCI reconstruction problem, which was previously dominated by traditional optimization methods.
  • Self-Attention for Spectral-Spatial Correlation: Integrating self-attention is crucial for capturing both local spatial details and global spectral relationships, leading to more accurate reconstructions.
  • Hierarchical Reconstruction: Breaking down the reconstruction of many spectral channels into a progressive, hierarchical approach is more effective than a single-shot reconstruction, especially with limited training data.
  • Two-Stage Approach: The combination of a powerful reconstruction stage and a lightweight refinement stage allows for both high-quality and efficient processing. The refinement stage, by processing channels independently, significantly speeds up the final output generation.

Key Results

  • PSNR Improvement: λ-net achieved a significant improvement of over 4dB in PSNR compared to the state-of-the-art DeSCI algorithm on simulation data. On real data, it also showed better performance.
  • Speedup: Reconstruction time was drastically reduced from hours (DeSCI) to sub-seconds (under 1 second on a CPU, and as fast as 33 milliseconds on a NVIDIA GTX 1080 Ti GPU). This represents a speedup of over 1000 times compared to DeSCI.
  • Qualitative Improvements: λ-net successfully recovers fine details, such as continuous wires, which are often lost or smoothed out by other methods like DeSCI. It also provides more accurate spectral curves.
  • Component Importance: Ablation studies showed that all components of λ-net (GAN, self-attention, HCR, refinement stage, residual learning, and using masks) contribute significantly to its performance.

Limitations and Future Work

  • Training Data: The paper acknowledges the challenge of limited training datasets for SCI. While spectral interpolation and data augmentation were used, the availability of diverse, large-scale SCI datasets remains a challenge.
  • Real-Mask-in-the-Loop (MIL): While MIL simulation is a good approach, it still relies on generated measurements from real masks, not entirely end-to-end real-world capture and reconstruction.
  • Future Work: The authors envision using λ-net with CASSI cameras to build an end-to-end video-rate 3D hyperspectral imaging system. They also suggest that joint training of the reconstruction and refinement stages might be possible with sufficient computational resources.

Summary updated

BibTeX
@inproceedings{miao2019l,
  author = {Xin Miao and Xin Yuan and Yunchen Pu and Vassilis Athitsos},
  title = {{l-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
}
Figure