Robust single-shot 3D fluorescence imaging in scattering media with a simulator-trained neural network.

Jeffrey Alido, Joseph Greene, Yujia Xue, Guorong Hu, Yunzhe Li, Mitchell Gilmore, Kevin J Monk, Brett T Dibenedictis, Ian G Davison, Lei TianOptics Express 2024 · OpenReview

TL;DR

This paper introduces SBR-Net, a deep learning model trained on synthetic data to reconstruct 3D fluorescence images from single-shot light-field measurements in scattering media. It achieves robust performance even with very low signal-to-background ratios (SBRs), outperforming traditional methods and demonstrating potential for deep tissue imaging.

What Problem Does This Paper Address?

Imaging deep within biological tissues is crucial for understanding cellular processes, but it's severely hampered by light scattering. Scattering causes signals to attenuate exponentially and creates a strong, heterogeneous background, drastically reducing the signal-to-background ratio (SBR). This makes it incredibly difficult to discern faint fluorescent signals from deeper structures. While light-field microscopy offers high-speed 3D imaging capabilities, the inherent ill-posed nature of 3D reconstruction from 2D light-field data is further exacerbated by scattering. Existing deep learning approaches often require extensive, experimentally acquired paired training data (ground truth and corresponding scattered images), which is costly and time-consuming to obtain. Furthermore, models trained on specific experimental conditions struggle to generalize to variations in scattering or background, leading to poor performance on "out-of-distribution" data.

Methods

The core of this work is the development of SBR-Net, a deep neural network designed for robust 3D fluorescence reconstruction from low-SBR light-field measurements in scattering media.

  • Simulator-Trained Approach: To overcome the challenge of acquiring paired training data, the authors developed a sophisticated scattering simulator. This simulator models the scattering-free imaging process of their Computational Miniature Mesoscope (CM²) and integrates a computationally efficient synthetic background model. This model captures the slow, spatially varying background characteristic of heterogeneous scattering.
  • Forward Model: The synthetic scattering measurement g(u, v) is generated by combining a free-space measurement PSF(u, v; z) * V(u, v; z) with a synthetic background BG, and then adding signal-dependent mixed Poisson-Gaussian (MPG) noise to mimic realistic experimental conditions.
  • SBR-Net Architecture: SBR-Net employs a dual-branch ResNet architecture. It takes the 3x3 views from the light-field measurement as input and fuses information from view synthesis and light-field refocusing. A variance stabilization strategy is incorporated to improve training convergence.
  • Training Strategy: The network is trained solely on synthetic data generated by the simulator, covering a range of SBRs (1.1 to 3.0) and scattering conditions. This approach aims to achieve generalization to real experimental data without requiring direct experimental ground truth.

Datasets

  • Synthetic Training Data: The authors generated 500 pairs of synthetic scattering measurements and their corresponding ground-truth 3D volumes. This data was created using their custom scattering simulator, incorporating varying SBRs and scattering conditions.
  • Experimental Data: The trained SBR-Net was evaluated on:
  • Scattering Phantoms: Controlled phantoms with different scattering lengths (72µm, 182µm, 279µm) and fluorescent beads.
  • Biological Samples: A 75µm thick fixed section of mouse brain tissue with fluorescently labeled neurons.
  • Benchmark Networks: For comparison, they also trained and evaluated:
  • FS-Net: A free-space network trained on synthetic free-space data.
  • BGR-Net: A network trained on background-removed scattering measurements.

Key Insights and Ideas

  • Synthetic Data is Sufficient: The most significant insight is that a deep learning model (SBR-Net) trained exclusively on synthetically generated data can generalize remarkably well to real experimental measurements, even those with significantly lower SBRs than seen during training. This bypasses the need for costly experimental ground truth.
  • SBR is Key to Robustness: The SBR of the training data critically impacts the model's robustness and accuracy. A balanced SBR range (1.1-3.0) in training data provides a good trade-off between depth penetration and hallucination artifacts in experimental data.
  • Bias-Variance Tradeoff in Data Quantity: The number of unique training pairs influences the bias-variance tradeoff. While more data can reduce bias, too much can lead to overfitting to synthetic data, resulting in poor generalization to experimental data. For this specific task (reconstructing spherical emitters and rejecting background), a moderate number of training pairs (500) proved optimal.
  • Beyond Simple Background Removal: Traditional background removal methods often discard low-SBR target signals along with the background. SBR-Net's strength lies in its ability to discriminate and retain these faint signals, even when they are barely distinguishable from the background.

Key Results

  • Robust Reconstruction at Low SBR: SBR-Net successfully reconstructs emitters in 3D from single-shot light-field measurements with SBRs as low as 1.05, achieving an F1 score of around 0.9 up to one scattering length deep.
  • Depth Penetration: The network can reconstruct emitters up to one scattering length deep, with performance decaying gracefully with depth. The imaging depth is inversely proportional to the scattering length.
  • Outperforms Baselines: SBR-Net significantly outperforms FS-Net (free-space reconstruction) and a background-removed network (BGR-Net) in terms of reconstruction accuracy and depth penetration, especially for low-SBR and deep emitters. FS-Net and BGR-Net often remove faint emitters or generate false positives.
  • Axial Resolution: SBR-Net consistently recovers emitters with 25µm axial resolution, a significant improvement over model-based algorithms that suffer from much worse axial elongation.
  • Generalization to Biological Samples: Pilot studies on fixed mouse brain slices showed promising results, demonstrating SBR-Net's ability to reconstruct and localize neurons with low contrast, outperforming baseline methods.

Limitations and Future Work

  • Optical Aberrations: The current simulator models scattering but does not account for optical aberrations caused by imaging optics or the sample itself. The authors suggest that future work should investigate learning these aberrations to further improve model robustness.
  • Low-Contrast Signal Modeling: The current model focuses on low-contrast signals. For broader applications, it might be necessary to incorporate more complex signal behaviors.
  • Refraction Effects: The phantom experiments revealed that refraction effects from the dome geometry can lead to axial localization inaccuracies, which are not explicitly modeled.
  • Real-time Performance: While inference is relatively fast (0.11s per volume on CPU, 0.022s on GPU), further optimization for real-time applications might be desirable.

Summary updated

BibTeX
@inproceedings{alido2024robust,
  author = {Jeffrey Alido and Joseph Greene and Yujia Xue and Guorong Hu and Yunzhe Li and Mitchell Gilmore and Kevin J Monk and Brett T Dibenedictis and Ian G Davison and Lei Tian},
  title = {{Robust single-shot 3D fluorescence imaging in scattering media with a simulator-trained neural network.}},
  booktitle = {Optics Express},
  year = {2024},
  url = {https://openreview.net/forum?id=yaKNY0V6T3},
}
Figure