Deep-learning-augmented computational miniature mesoscope.

Yujia Xue, Qianwan Yang, Guorong Hu, Kehan Guo, Lei TianOptica 2023 · OpenReview

TL;DR

This paper introduces CM² V2, a compact, single-shot 3D fluorescence microscope that uses deep learning to achieve high-resolution imaging over a wide field of view. By combining novel hardware with a sophisticated AI model, it overcomes the traditional trade-offs between FOV, resolution, and system size, paving the way for more accessible and powerful biological imaging.

What Problem Does This Paper Address?

Fluorescence microscopy is crucial for studying biological structures and dynamics. However, existing systems face a fundamental challenge: achieving a large field of view (FOV) while maintaining high resolution and keeping the system compact and affordable. Traditional microscopes struggle to balance these requirements. Miniaturized microscopes (miniscopes) offer portability but often sacrifice FOV or resolution. While computational imaging techniques like lightfield microscopy have shown promise, they can be complex, limited in FOV, or suffer from reduced contrast and signal-to-noise ratio (SNR), especially for imaging weak fluorescent signals. This paper aims to bridge this gap by developing a computational miniature mesoscope that delivers micron-scale resolution across centimeter-scale FOVs in a compact form factor.

Methods

The CM² V2 system builds upon its predecessor with significant hardware and computational advancements.

Hardware: * Hybrid Emission Filter: A new filter design improves imaging contrast by 5x, enhancing the ability to detect weaker signals. * 3D-Printed Freeform Collimator: This novel illuminator design increases excitation efficiency by 3x, providing more uniform and efficient illumination. * Upgraded Sensor: A back-side illuminated (BSI) CMOS sensor improves SNR and dynamic range.

Computation: * 3D Linear Shift-Variant (LSV) Model: A new, efficient model accurately characterizes the spatially varying aberrations inherent in the CM² system across a large volume. This model is crucial for generating realistic training data. * CM2Net (Deep Learning Model): This multi-module neural network is trained using the 3D-LSV simulator. It comprises: * View Demixing Module: Separates the multiplexed image into individual views, exploiting unique aberration features. * View Synthesis Module: Reconstructs 3D information using disparity cues from the demixed views. * Lightfield Refocusing Enhancement Module: Further refines the reconstruction by removing artifacts and enhancing resolution.

The system's computational pipeline is designed to be fast and efficient, enabling real-time or near-real-time 3D reconstruction.

Datasets

  • Synthetic Data: The primary dataset for training CM2Net was generated using the authors' novel 3D-LSV simulator. This simulator creates synthetic volumes with varying FOVs, depths, emitter densities, sizes, and intensities, mimicking realistic biological scenarios. This approach bypasses the need for extensive experimental data collection.
  • Experimental Data:
    • Phantom Objects: Fluorescent beads (10-µm and 15-µm) embedded in resin were used for hardware calibration and experimental validation.
  • Biological Data (Simulated): The authors also simulated neuronal populations and mouse brain vasculature based on existing open-source datasets to demonstrate CM²Net's applicability to complex biological structures.

Key Insights and Ideas

  • The Power of Simulation for Training: The success of CM2Net hinges on its training with a highly accurate 3D-LSV simulator. This approach allows for the generation of diverse and realistic training data, overcoming the limitations of collecting vast amounts of experimental data, especially for complex or rare scenarios.
  • Decomposition of a Complex Problem: Breaking down the challenging 3D reconstruction task into distinct modules (view demixing, view synthesis, and enhancement) makes the problem more tractable for deep learning and allows for specialized optimization of each stage.
  • Freeform Optics and 3D Printing Synergy: The use of 3D-printed freeform optics for the illuminator demonstrates a cost-effective and accessible pathway to advanced optical system design, potentially democratizing high-performance imaging.
  • Axial Super-Resolution: The 3D-LSV model, with its axial interpolation, achieves axial "super-resolution" beyond the physical sampling of the PSF calibration, leading to significantly improved axial resolution.

Key Results

  • Resolution: Achieves ~6-µm lateral and ~25-µm axial resolution.
  • FOV and Depth: Reconstructs 3D images across a ~7-mm FOV and 800-µm depth.
  • Speed: Reconstruction time is <4 seconds on a standard GPU, a ~1400x speedup compared to previous model-based algorithms.
  • Axial Resolution Improvement: ~8x better axial resolution compared to model-based reconstruction.
  • Contrast Improvement: Hybrid emission filter improves image contrast by 5x.
  • Excitation Efficiency: 3D-printed illuminator achieves ~80% efficiency, a ~3x improvement.
  • Detection Performance: CM2Net achieves high recall (~0.78) and precision (~0.80) in experiments, comparable to state-of-the-art deep learning methods.
  • Localization Accuracy: Averaged lateral and axial RMSEs are 4.17 µm and 11.2 µm, close to single-voxel accuracy.
  • Robustness: Demonstrates robustness to variations in emitter size, density, depth, and intensity, and generalizes well to experimental data and complex structures like neuronal populations and vasculature.

Limitations and Future Work

  • Peripheral FOV Degradation: While CM²Net performs well across the FOV, recall slightly degrades at the extreme edges. This is attributed to imperfect view-demixing, potentially caused by the rapid intensity fall-off of the point source used for PSF calibration and extra views from partial microlenses.
  • Discontinuity Artifacts in Dense Objects: When reconstructing dense objects like vasculature, discontinuity artifacts appear. This is due to the sparsity prior enforced by training on particle datasets. The authors suggest fine-tuning CM2Net on more complex datasets to address this.
  • Tissue Scattering: Tissue scattering remains a challenge for CM² and is an area for future investigation, with potential solutions including miniature structured illumination and scattering-incorporated reconstruction frameworks.
  • Miniaturization: While compact, the system size is currently limited by the availability of miniature BSI CMOS sensors. Future advancements in sensor technology could lead to even smaller, "wearable" systems.
  • Freeform Optics Integration: The authors envision further exploration of 3D-printed freeform optics for future CM² platforms and other miniature optical devices.

Summary updated

BibTeX
@inproceedings{xue2023deep,
  author = {Yujia Xue and Qianwan Yang and Guorong Hu and Kehan Guo and Lei Tian},
  title = {{Deep-learning-augmented computational miniature mesoscope.}},
  booktitle = {Optica},
  year = {2023},
  url = {https://openreview.net/forum?id=SJsbh7KBWz},
}
Figure