Model-Inspired Deep Learning for Light-Field Microscopy with Application to Neuron Localization

Pingfan Song, Herman Verinaz-Jadan, Carmel L. Howe, Peter Quicke, Amanda J. Foust, Pier Luigi Dragotti

Abstract

Capturing the 3D dynamics of neuronal networks is a cornerstone of modern neuroscience. Light-field microscopy (LFM) offers a promising solution by capturing volumetric information in a single snapshot. However, reconstructing precise 3D coordinates from raw LFM data is computationally intensive. We introduce CISTA-net, a deep learning framework that "unrolls" the Iterative Shrinkage and Thresholding Algorithm (ISTA) into a neural network. This model-inspired approach combines the interpretability of physics-based models with the speed of deep learning, achieving a 10,000x speedup over traditional methods and superior localization accuracy.

The Challenge of 3D Neuron Localization

Light-field microscopy (LFM) utilizes an array of microlenses to capture both spatial and angular information of light rays. While powerful, the resulting images are often blurry and require complex mathematical reconstruction to yield 3D data. Traditional model-based methods, such as 3D deconvolution or sparse coding, are accurate but slow, often taking seconds per frame. Conversely, standard deep learning "black boxes" lack physical grounding and require immense labeled datasets.

Figure 1: Light-field microscopy (LFM) imaging principles and the generation of Epipolar Plane Images (EPIs).

The authors address this by focusing on Epipolar Plane Images (EPIs)—2D slices of light-field data that encapsulate depth information through the slope of signal lines. As shown in Figure 1, different imaging situations (native focal plane, above, or below) result in distinct patterns in the EPI space.

Methodology: CISTA-net and Algorithm Unrolling

The core innovation, CISTA-net, is built using Algorithm Unrolling. The researchers took the classical ISTA optimization algorithm, used for solving Convolutional Sparse Coding (CSC) problems, and mapped its iterations directly onto neural network layers.

Figure 2: The architecture of CISTA-net, derived by unrolling the ISTA optimization algorithm.

In this architecture, depicted in Figure 2:

  • Physics-Grounded Design: The network's structure, including skip connections and specific convolutions, is derived from the mathematical steps of ISTA rather than heuristic design.
  • Soft-Label Training: To improve depth resolution, the model was trained using "soft-labels"—ground-truth depth data smoothed with a Gaussian kernel—allowing the network to better distinguish between neurons at nearly identical depths.

Experimental Setup

The model was trained on a dataset of 100,000 simulated EPI samples. For real-world validation, the authors evaluated CISTA-net on 55 mammalian neuron specimens (genetically encoded fluorophores in mouse brain tissue) across a depth range of -18 to 36 $\mu$m.

Key Results

CISTA-net demonstrates significant improvements in both efficiency and precision:

  • Unprecedented Speed: The network processes an EPI in just 0.216 milliseconds, a 10,000x speedup over traditional CSC methods (~3.6 seconds). This performance enables potential real-time 3D tracking.
  • Enhanced Accuracy: CISTA-net achieved a Z-axis Root Mean Square Error (RMSE) of 0.82 $\mu$m, outperforming the best traditional sparse coding approaches (1.14 $\mu$m).
  • Robust Localization: The model produces cleaner sparse codes with fewer artifacts, facilitating more reliable neuron detection in complex volumes.

Discussion and Future Work

By treating neuron localization as a sparse coding problem, CISTA-net effectively filters noise and focuses on point-like signal sources. While currently optimized for individual neurons, future iterations could adapt the model for more complex structures like dense dendritic trees or deeper, more scattering brain tissue. Integrating the raw-to-EPI conversion directly into the pipeline remains a promising avenue for further streamlining the reconstruction process.

Summary updated

Abstract

Light-field microscopes are able to capture spatial and angular information of incident light rays. This allows reconstructing 3D locations of neurons from a single snap-shot. In this work, we propose a model-inspired deep learning approach to perform fast and robust 3D localization of sources using light-field microscopy images. This is achieved by developing a deep network that efficiently solves a convolutional sparse coding (CSC) problem to map Epipolar Plane Images (EPI) to corresponding sparse codes. The network architecture is designed systematically by unrolling the convolutional Iterative Shrinkage and Thresholding Algorithm (ISTA) while the network parameters are learned from a training dataset. Such principled design enables the deep network to leverage both domain knowledge implied in the model, as well as new parameters learned from the data, thereby combining advantages of model-based and learning-based methods. Practical experiments on localization of mammalian neurons from light-fields show that the proposed approach simultaneously provides enhanced performance, interpretability and efficiency.
BibTeX
@inproceedings{song2021model,
  author = {Pingfan Song and Herman Verinaz-Jadan and Carmel L. Howe and Peter Quicke and Amanda J. Foust and Pier Luigi Dragotti},
  title = {{Model-Inspired Deep Learning for Light-Field Microscopy with Application to Neuron Localization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=DJ45Z4OGgp},
}
Figure