Light-field deep learning enables high-throughput, scattering-mitigated calcium imaging

Carmel L. Howe, K. Y. Zhao, Herman Verinaz-Jadan, Pingfan Song, Samuel J. Barnes, Pier Luigi Dragotti, Amanda J. Foust

TL;DR

This paper introduces 2PiLnet, a deep learning framework that uses light-field microscopy (LFM) to achieve high-speed, volumetric calcium imaging in scattering brain tissue. By combining LFM's scanless nature with a physics-informed neural network, 2PiLnet reconstructs neural activity with two-photon-like contrast and reduced scattering artifacts, enabling real-time analysis for complex neuroscience experiments.

What Problem Does This Paper Address?

Understanding neural circuit function requires observing the activity of many neurons simultaneously over time. Traditional methods like two-photon (2P) microscopy offer high-quality imaging but are slow due to scanning. Light-field microscopy (LFM) offers a solution by capturing entire volumes in a single snapshot, dramatically increasing speed and throughput. However, LFM suffers from two major drawbacks: its high computational cost for volume reconstruction and its susceptibility to light scattering in biological tissues, which degrades image quality and signal-to-noise ratio (SNR). These limitations have hindered LFM's widespread adoption for real-time, adaptive experiments that demand fast, reliable neural activity data from scattering environments like the mammalian brain.

Methods

The core innovation is the 2PiLnet deep learning framework. It's a physics-based neural network designed to reconstruct high-quality volumetric calcium signals from one-photon LFM data, even in the presence of scattering.

Here's a breakdown of the approach:

  • Dual-Mode Microscopy: The system combines a standard two-photon (2P) laser scanning microscope with a light-field microscope (LFM). This allows for acquiring both high-contrast 2P data (for training) and fast, scanless LFM data (for actual experiments).
  • 2PiLnet Architecture: This physics-informed neural network is trained using a combination of 2P volumes and one-photon light fields. It integrates the LFM wave-optics forward model into its training process, ensuring more reliable volume estimation.
  • Training Strategy: 2PiLnet undergoes a two-phase training:
  • Supervised Training: Uses paired 2P z-stacks and corresponding LFM z-stacks of structural markers (tdTomato) to learn the relationship between scattered light-field data and high-quality 2P volumes.
  • Self-Supervised Training: Leverages LF videos of calcium indicators (jGCaMP8f) and the LFM forward model to refine volume estimation without requiring additional 2P data for every new field of view.
  • Calcium Signal Extraction:
    • Preprocessing: Raw LF videos are processed to create an "LF activity map" that highlights active neurons with high contrast.
  • Segmentation: The 2PiLnet-estimated volume is segmented into regions of interest (ROIs) using Voronoi-Otsu labeling.
  • Time Series Calculation: Two methods are used:
  • 2PiLnet ROI: Calculates the mean intensity within each ROI over time.
  • 2PiLnet Matrix: Uses a CNN to estimate each neuron's "footprint" in the light field, forming a matrix that's used with the LF video to compute time series via matrix factorization.

Datasets

  • Training Data:
    • Paired 2P z-stacks and one-photon LFM z-stacks of tdTomato-expressing neurons in mouse neocortical brain slices.
  • One-photon LFM videos of jGCaMP8f-expressing neurons in mouse neocortical brain slices.
  • Evaluation Data:
    • Light-field videos of jGCaMP8f-expressing neurons in neocortical brain slices, acquired at 100 Hz.
  • Simulated LF videos of neuronal calcium activity corrupted by scattering and noise (for assessing precision and recall).

The paper does not introduce a new benchmark dataset but rather uses existing experimental setups and data types for training and evaluation.

Key Insights and Ideas

  • Bridging the Gap: 2PiLnet successfully bridges the gap between the speed of LFM and the quality of 2P imaging, offering a path to high-throughput, scattering-robust neural circuit analysis.
  • Physics-Informed AI: Integrating the physics of LFM (wave-optics forward model) into the deep learning architecture is crucial for reliable volume reconstruction and generalization to unseen data.
  • Generalization Power: The trained 2PiLnet model can generalize to new fields of view without requiring new 2P ground truth data, significantly reducing experimental overhead.
  • Scattering Mitigation: The network effectively reconstructs volumes with two-photon-like contrast and source confinement, even from scattered, blurry one-photon LFM inputs.
  • Real-time Potential: The dramatic reduction in processing time (hours to minutes) brings volumetric LFM analysis within reach of single-workstation, real-time applications.

Key Results

  • High-Speed Imaging: Achieved volumetric imaging at 100 volumes per second.
  • Scattering Mitigation: Reconstructed volumes with two-photon-like contrast and source confinement from scattered LFM data.
  • High SNR: Extracted calcium signals with high signal-to-noise ratios (median SNR ~13 dB for 2PiLnet Matrix and ROI).
  • Reduced Crosstalk: Significantly reduced optical crosstalk between adjacent neurons compared to conventional methods like Richardson-Lucy (RL) deconvolution and seeded-iterative demixing (SID). For example, the correlation coefficient between adjacent cells was reduced from 0.60 (RL) to 0.41 (2PiLnet ROI).
  • Spike Detection: Enabled the detection of putative spikes fired at rates up to 10 Hz.
  • Computational Speedup: Reduced LF video processing time from hours to minutes, an ~18-fold reduction compared to SID for the 2PiLnet Matrix workflow.
  • Neuron Ensemble Mapping: Demonstrated the ability to map spatially intermingled neuronal ensembles with correlated activity.

Limitations and Future Work

  • Depth Limitations: The study primarily imaged the superficial 100 microns of brain slices. Future work will assess 2PiLnet's performance at greater depths in vivo.
  • Motion Artifacts: In vivo imaging will require correction for motion artifacts, which are not present in the in vitro slice preparations.
  • Generalizability to Different Optics/Samples: While 2PiLnet generalizes to unseen FOVs, further characterization is needed for its performance with different LFM optics, system-to-system alignment variations, and samples with varying indicator sparsities and depths.
  • Hardware Advancements: The camera's maximum frame rate currently limits the volume rate. Faster, more sensitive cameras could enable even higher imaging speeds.
  • Real-time Processing: While processing is significantly faster, achieving true real-time signal extraction for adaptive experiments may require further optimization, potentially through GPU or FPGA architectures.
  • Training Data Requirements: While 2PiLnet requires only a single 2P volume for initial training, future work could explore training with even less data or adapting to different microscope configurations without 2P data.

Summary updated

Abstract

Light-field microscopy (LFM) enables high-throughput functional imaging by scanlessly encoding entire volumes in single snapshots. However, LFM's computational burden and vulnerability to scattering limit its application to biological imaging. We present a light-field strategy for volumetric, scattering-mitigated neural circuit activity monitoring. A physics-based deep neural network, 2PiLnet, is trained with two-photon volumes and one-photon light fields. Light-field videos of jGCaMP8f-expressing neurons are acquired in neocortical brain slices. 2PiLnet reconstructs volumes with two-photon-like contrast and source confinement from scattered, blurry one-photon light fields from fields-of-view for which no two-photon images are provided. This enables automated segmentation and extraction of calcium signals with high signal-to-noise ratios and reduces optical crosstalk compared to conventional volume reconstruction methods. Imaging 100 volumes per second, we observe putative spikes fired at up to 10 Hz and the spatial intermingling of putative ensembles throughout [Formula: see text]-micron volumes. Compared to iterative algorithms, 2PiLnet workflows reduce light-field video processing times by several-fold, advancing the goal of real-time, scattering-robust volumetric neural circuit imaging for closed-loop and adaptive experimental paradigms.
BibTeX
@inproceedings{howe2025light,
  author = {Carmel L. Howe and K. Y. Zhao and Herman Verinaz-Jadan and Pingfan Song and Samuel J. Barnes and Pier Luigi Dragotti and Amanda J. Foust},
  title = {{Light-field deep learning enables high-throughput, scattering-mitigated calcium imaging}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2025},
  doi = {10.1073/pnas.2510337122},
}
Figure