Hybrid Space Calibrated 3D Network of Diffractive Hyperspectral Optical Imaging Sensor

Hao Fan, Chenxi Li, Bo Gao, Huangrong Xu, Yuwei Chen, Xuming Zhang, Xu Li, Weixing YuSensors 2024 · DOI

TL;DR

This paper introduces "HSC3D," a novel hybrid deep learning network for diffractive multispectral imaging. It addresses the challenge of spatially varying image distortions (magnification and rotation) inherent in diffractive optics, significantly improving the quality of reconstructed multispectral images.

What Problem Does This Paper Address?

Diffractive multispectral imaging is a powerful technique for analyzing materials and environments by capturing spectral information. However, a major hurdle is the "spatially variant point spread function" (PSF). This means the way light is blurred and spread across the sensor isn't uniform; it changes depending on the location in the image and the wavelength of light. This variation is primarily caused by magnification and rotation effects that differ across wavelengths and spatial positions. These distortions lead to blurry and inaccurate multispectral images, limiting the precision and applicability of the technique. Existing methods often require complex hardware modifications, extensive calibration procedures, or large datasets, making them impractical for many real-world scenarios.

Methods

The core of this work is the "Hybrid Space Calibrated 3D Network" (HSC3D). This approach combines two key components:

  • Space Calibration Module: This module is designed to tackle the magnification and rotation distortions. It uses an intensity-based automatic image registration method to identify features in the multispectral images. By analyzing these features, it calculates and corrects for the varying magnification and rotation effects across different wavelengths. This calibration is performed before feeding the data into the deep learning network.
  • 3D U-Net Architecture: The calibrated multispectral data is then processed by a 3D U-Net. This deep learning model, built with 3D convolutional layers, is adept at learning complex spatial and spectral relationships. The U-Net architecture, with its encoder-decoder structure, effectively captures features at different scales and reconstructs high-quality multispectral images. The paper also mentions using Google's MAXIM model for an initial denoising step.

The system also involves a custom-designed diffractive multispectral imaging system with a diffractive lens and four refractive lenses, where monochromatic aberrations are corrected to ensure high initial imaging quality.

Datasets

The paper does not explicitly mention using publicly available datasets for training or evaluation. Instead, it appears to rely on:

  • Simulated Multispectral Data: The authors simulate multispectral data through the forward process of diffractive optical imaging.
  • Measured Experimental Data: They also use data acquired from their fabricated diffractive multispectral imaging system.

The paper does not introduce a new benchmark dataset but rather evaluates its method on data generated from its experimental setup.

Key Insights and Ideas

  • Decoupling Calibration and Reconstruction: A significant insight is the separation of spatial distortion correction (magnification/rotation) into a dedicated calibration module before the deep learning reconstruction. This allows the 3D U-Net to focus on spectral restoration without being overwhelmed by geometric inaccuracies.
  • Leveraging 3D Convolutions for Spectral Data: The use of 3D U-Net is crucial. Unlike 2D networks that treat spectral channels independently, 3D convolutions can effectively learn correlations across spectral channels, which is vital for accurate multispectral reconstruction.
  • Practicality without Hardware Overhaul: The HSC3D approach aims to improve image quality without requiring complex and expensive modifications to the imaging hardware, making it more accessible.
  • Addressing Spatially Variant PSF is Key: The paper strongly emphasizes that the spatially variant nature of the PSF is the primary challenge, and their method directly targets this by combining geometric calibration with advanced deep learning.

Key Results

The HSC3D algorithm demonstrated significant improvements in image quality:

  • Quantitative Improvements:
    • Mean Peak Signal-to-Noise Ratio (MPSNR) improved by 3.33 dB.
  • Mean Structural Similarity Index (MSSIM) improved by 0.08.
  • These improvements are compared to a typical "Unrolled Network" algorithm.
  • Qualitative Improvements:
    • Reconstructed multispectral images from HSC3D are described as "clearer" and avoid the "ghosting issues" observed in the Unrolled Network.
  • The method shows better performance across all eight spectral channels, particularly at longer wavelengths where distortions are more pronounced.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the current system is limited to eight spectral channels.
  • While the method reduces space-variant issues, it doesn't explicitly address other complex PSF variations like aberrations from different optical elements.
  • The current calibration is focused on magnification and rotation; other spatial variants might still pose challenges.
  • Future Work:
    • Extending the system to achieve an even wider spectrum and more sensitive diffractive multispectral imaging capabilities.
  • Investigating and calibrating more complex PSF variations, such as aberrations from different optical elements.
  • Exploring a "grid hybrid space calibrated 3D algorithm" for discrete pixel-by-pixel PSF estimation to advance diffractive multispectral imaging for applications like space imaging, micrography, and security cameras.

Summary updated

Abstract

Diffractive multispectral optical imaging plays an essential role in optical sensing, which typically suffers from the image blurring problem caused by the spatially variant point spread function. Here, we propose a novel high-quality and efficient hybrid space calibrated 3D network "HSC3D" for spatially variant diffractive multispectral imaging that utilizes the 3D U-Net structure combined with space calibration modules of magnification and rotation effects to achieve high-accuracy eight-channel multispectral restoration. The algorithm combines the advantages of the space calibrated module and U-Net architecture with 3D convolutional layers to improve the image quality of diffractive multispectral imaging without the requirements of complex equipment modifications and large amounts of data. A diffractive multispectral imaging system is established by designing and manufacturing one diffractive lens and four refractive lenses, whose monochromatic aberration is carefully corrected to improve imaging quality. The mean peak signal-to-noise ratio and mean structural similarity index of the reconstructed multispectral images are improved by 3.33 dB and 0.08, respectively, presenting obviously improved image quality compared with a typical Unrolled Network algorithm. The new algorithm with high space calibrated ability and imaging quality has great application potential in diffraction lens spectroscopy and paves a new method for complex practical diffractive multispectral image sensing.
BibTeX
@inproceedings{fan2024hybrid,
  author = {Hao Fan and Chenxi Li and Bo Gao and Huangrong Xu and Yuwei Chen and Xuming Zhang and Xu Li and Weixing Yu},
  title = {{Hybrid Space Calibrated 3D Network of Diffractive Hyperspectral Optical Imaging Sensor}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24216903},
}
Figure