Deep learning approach for hyperspectral image demosaicking, spectral correction and high-resolution RGB reconstruction

Peichao Li, Michael Ebner, Philip J. Noonan, Conor C. Horgan, Anisha Bahl, Sébastien Ourselin, Jonathan Shapey, Tom Vercauteren

TL;DR

This paper introduces a deep learning approach to reconstruct high-quality hyperspectral images from snapshot mosaic cameras, overcoming the limitations of current real-time surgical imaging. By simulating realistic snapshot data from high-resolution sources and employing a novel U-Net architecture, they achieve fast and accurate spectral and spatial reconstruction, paving the way for improved intraoperative diagnostics.

What Problem Does This Paper Address?

Intraoperative hyperspectral imaging (HSI) holds immense promise for real-time tissue characterization during surgery, enabling better discrimination between healthy and diseased tissues. However, current HSI systems often suffer from slow acquisition times, making them unsuitable for live surgical guidance. Snapshot mosaic cameras offer a solution by capturing hyperspectral data in a single exposure, but they produce an "ill-posed" problem: the raw data is spatially undersampled and spectrally mixed. This requires complex demosaicking and spectral correction steps to recover meaningful information, which have historically been challenging to perform in real-time with sufficient accuracy. This paper aims to bridge this gap by developing a fast and accurate deep learning-based method for demosaicking and reconstructing hyperspectral images from snapshot mosaic cameras for surgical applications.

Methods

The core of this work lies in a two-part framework: synthetic data generation and a supervised deep learning approach.

  1. Synthetic Data Generation: Since real-world hyperspectral data from snapshot mosaic cameras in surgical settings is scarce, the authors created a synthetic dataset. They leveraged publicly available high-resolution hyperspectral line-scan datasets (HELICoiD and ODSI-DB).
  2. Spectral Response Simulation: They simulated the spectral response of a snapshot mosaic sensor, accounting for realistic effects like harmonics, cross-talk, and spectral leakage, to generate intermediate hyperspectral cubes with a reduced number of spectral bands.
  3. Spatial Subsampling: They then simulated the spatial undersampling inherent in snapshot mosaic cameras by dividing the hyperspectral cube into tiles and selecting only one pixel per tile, mimicking the 2D mosaic output.
  4. Target Ideal Hyperspectral Data: They also generated "ideal" hyperspectral cubes, representing the ground truth, by applying ideal Lorentzian spectral responses to the high-resolution data. This serves as the target for their learning algorithm.
  5. Deep Learning for Demosaicking and Reconstruction:
    • U-Net Architecture: They employed a residual U-Net architecture, known for its effectiveness in image super-resolution and detail refinement. This network is designed to take the bilinearly interpolated snapshot mosaic image as input.
  6. Supervised Learning: The network is trained using the synthetically generated pairs of snapshot mosaic images and their corresponding ideal hyperspectral cubes.
  7. Spectral Correction: The output of the U-Net (which is spectrally corrected and spatially super-resolved) is further refined by applying a camera-specific calibration matrix to compensate for parasitic spectral effects.
  8. Loss Functions: They explored two loss configurations: L1 loss and a perceptual loss (using VGG-16 features). The perceptual loss was found to yield better results.
  9. sRGB Reconstruction: For visualization and qualitative evaluation, the reconstructed hyperspectral data is converted into standard RGB images.

Datasets

  • Training and Validation:
    • HELICoiD dataset: Publicly available hyperspectral cubes acquired during neurosurgical procedures (400-1000 nm, 826 spectral bands). Used for training, validation, and testing.
  • ODSI-DB dataset: Publicly available oral and dental hyperspectral images. Specifically, data from a Specim IQ line-scan camera (400-1000 nm, 204 spectral bands) was used for training and testing.
  • Real Data Evaluation: A real snapshot mosaic image from a Ximea xiSpec camera was used for preliminary validation of the algorithm's generalizability.

Key Insights and Ideas

  • Synthetic Data as a Powerful Proxy: The ability to generate realistic synthetic hyperspectral snapshot data from existing high-resolution datasets is crucial for training deep learning models when real-world surgical data is limited. This approach effectively overcomes the "data scarcity" problem.
  • U-Net for Hyperspectral Super-Resolution: Adapting the U-Net architecture, which has shown success in natural image super-resolution, to the hyperspectral domain proves effective for recovering lost spatial detail.
  • Perceptual Loss for Better Visual Quality: Using a perceptual loss function, which focuses on feature similarity rather than pixel-wise differences, leads to perceptually more pleasing and sharper reconstructed images, especially important for visual interpretation in surgery.
  • Separation of Spatial and Spectral Tasks: The framework separates the spatial super-resolution task (handled by the U-Net) from spectral correction (handled by a calibration matrix), simplifying the learning process and leveraging existing sensor knowledge.

Key Results

  • Quantitative Improvement: The proposed demosaicking algorithm, particularly with the perceptual loss, significantly outperforms linear demosaicking in terms of SSIM and PSNR metrics on both HELICoiD and ODSI-DB datasets. For example, SSIM scores improved from around 0.932 to 0.975 on the HELICoiD dataset.
  • Qualitative Improvement: Visual comparisons (Figure 3) show that the deep learning approach recovers finer details, such as vessels, that are lost in linear demosaicking. The perceptual loss model produces sharper images.
  • User Study Validation: A user study with 12 clinical experts rated the demosaicked images. The proposed algorithm (perceptual loss) received significantly higher subjective quality scores (average 3.08) compared to linear demosaicking (average 1.14), though still below the ideal (average 3.60).
  • Real-time Performance: The algorithm achieves a processing time of approximately 45 ms per frame on a high-end workstation, making it suitable for real-time surgical applications. The U-Net inference itself takes only 34 ms.
  • Cross-Dataset Generalization: The model trained on ODSI-DB showed reasonable performance when tested on HELICoiD, indicating some degree of generalizability, although cross-dataset results were slightly worse than within-dataset training.

Limitations and Future Work

  • Domain Gap: The authors acknowledge a domain gap between the HELICoiD and ODSI-DB datasets, which can lead to artefacts when using models trained on one dataset for the other without fine-tuning.
  • Synthetic Data Reliance: While effective, the reliance on synthetic data means that performance on truly novel or unexpected real-world scenarios might still be a concern.
  • Further Real Data Investigation: Future work will focus on further investigating the generalisability of the algorithm with more real snapshot data.
  • Speed and Quality Improvements: There is still room for improvement in both processing speed and image quality.
  • Integration into Surgical Systems: The authors aim for seamless integration into real-time surgical hyperspectral imaging systems for responsive surgical guidance.

Summary updated

Abstract

Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections to recover spatial and spectral information of the image. In this work, we propose a deep learning-based image demosaicking algorithm for snapshot hyperspectral images using supervised learning methods. Due to the lack of publicly available medical images acquired with snapshot mosaic cameras, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets captured by high-resolution, but slow, hyperspectral imaging devices. Image reconstruction is achieved using convolutional neural networks for hyperspectral image super-resolution, followed by cross-talk and leakage correction using a sensor-specific calibration matrix. The resulting demosaicked images are evaluated both quantitatively and qualitatively, showing clear improvements in image quality compared to a baseline demosaicking method using linear interpolation. Moreover, the fast processing time of~45\,ms of our algorithm to obtain super-resolved RGB or oxygenation saturation maps per image frame for a state-of-the-art snapshot mosaic camera demonstrates the potential for its seamless integration into real-time surgical hyperspectral imaging applications.
BibTeX
@inproceedings{li2022deep,
  author = {Peichao Li and Michael Ebner and Philip J. Noonan and Conor C. Horgan and Anisha Bahl and Sébastien Ourselin and Jonathan Shapey and Tom Vercauteren},
  title = {{Deep learning approach for hyperspectral image demosaicking, spectral correction and high-resolution RGB reconstruction}},
  booktitle = {Comput. Methods Biomech. Biomed. Eng. Imaging Vis},
  year = {2022},
  url = {https://openreview.net/forum?id=OiInA7ArdP},
}
Figure