Fluctuation-based deconvolution in fluorescence microscopy using plug-and-play denoisers

Vasiliki Stergiopoulou, Subhadip Mukherjee, Luca Calatroni, Laure Blanc-Féraud

Abstract

The spatial resolution of images of living samples obtained by fluorescence microscopes is physically limited due to the diffraction of visible light, which makes the study of entities of size less than the diffraction barrier (around 200 nm in the x-y plane) very challenging. To overcome this limitation, several deconvolution and super-resolution techniques have been proposed. Within the framework of inverse problems, modern approaches in fluorescence microscopy reconstruct a super-resolved image from a temporal stack of frames by carefully designing suitable hand-crafted sparsity-promoting regularisers. Numerically, such approaches are solved by proximal gradient-based iterative schemes. Aiming at obtaining a reconstruction more adapted to sample geometries (e.g. thin filaments), we adopt a plug-and-play denoising approach with convergence guarantees and replace the proximity operator associated with the explicit image regulariser with an image denoiser (i.e. a pre-trained network) which, upon appropriate training, mimics the action of an implicit prior. To account for the independence of the fluctuations between molecules, the model relies on second-order statistics. The denoiser is then trained on covariance images coming from data representing sequences of fluctuating fluorescent molecules with filament structure. The method is evaluated on both simulated and real fluorescence microscopy images, showing its ability to correctly reconstruct filament structures with high values of peak signal-to-noise ratio (PSNR).

Introduction

In optical microscopy, the diffraction limit serves as a fundamental physical barrier, preventing the observation of biological structures smaller than approximately 200 nm. To resolve sub-cellular details, researchers have developed super-resolution techniques that exploit the temporal fluctuations—the "flickering"—of fluorescent molecules. By analyzing a temporal stack of images, mathematical deconvolution can be used to recover information beyond the diffraction limit.

Traditional inverse-problem approaches typically rely on hand-crafted regularizers. These are mathematical assumptions, such as sparsity, that guide the reconstruction process. While effective for isolated point sources, these priors often struggle with continuous biological features like the cytoskeleton. In such cases, traditional methods frequently produce "dotted" artifacts, reconstructing smooth filaments as series of disconnected points. This paper introduces PnP-COL0RME, a framework designed to overcome these limitations by combining temporal fluctuation statistics with "Plug-and-Play" (PnP) deep learning denoisers.

Methodology

Covariance-Based Modeling

Rather than processing raw image frames directly, PnP-COL0RME operates on the auto-covariance (second-order statistics) of the temporal stack. This approach is advantageous because the effective point spread function (PSF) in the covariance domain is squared, resulting in a narrower profile and naturally higher spatial resolution. This statistical foundation accounts for the independence of fluctuations between individual molecules.

The Plug-and-Play (PnP) Framework

In classical optimization, a "proximal operator" is used to enforce a specific prior, such as $\ell_1$ sparsity. The PnP framework replaces this rigid mathematical step with a pre-trained deep learning denoiser. This allows the model to utilize an implicit prior learned from data rather than an explicit formula defined by a researcher.

For the denoiser, the authors utilize DRUNet, a Convolutional Neural Network (CNN). Crucially, the network is not trained to solve the entire deconvolution problem; instead, it is trained specifically to denoise covariance images of filament structures. This specialization allows the network to become "aware" of the specific geometric properties of filaments.

Convergence and Stability

To address the "black box" nature of deep learning, the authors employ a specific gradient-step formulation for the denoiser. This ensures that the iterative algorithm maintains rigorous convergence guarantees, preventing the hallucinations or oscillations that can occur in unconstrained neural networks.

Experimental Results

Datasets

The method was evaluated using both simulated and real-world data:

  • Simulated Data: Based on the SMLM 2016 challenge (MT0 microtubule set), the authors generated 500 pairs of clean and noisy auto-covariance images, simulating 500 frames of temporal fluctuations with added Gaussian noise.
  • Real Data: The model was tested on high-density acquisitions from the 2013 SMLM challenge, featuring complex and overlapping tubulin structures.

Performance

Quantitatively, PnP-COL0RME achieved a Jaccard Index (JI) of 0.67 for support localization and a PSNR of 26.01 on simulated data. Qualitatively, the method demonstrated a significant improvement in "continuation properties." Unlike traditional $\ell_0$ or $\ell_1$ methods that produce punctuated lines, PnP-COL0RME successfully reconstructed smooth, continuous curvilinear structures in both simulated and real-world datasets.

Discussion and Future Work

By decoupling the physics of the microscope (the forward operator) from the image prior (the denoiser), PnP-COL0RME offers a flexible architecture. If the microscope's blur characteristics change, the physics component can be updated without retraining the neural network.

However, some limitations remain. The current model utilizes only the diagonal of the covariance matrix (auto-covariance), ignoring cross-pixel correlations that could provide additional information. Furthermore, the current iteration does not explicitly model the downsampling from a high-resolution grid to a low-resolution camera sensor. Future work will focus on integrating these cross-terms and downsampling operators to further enhance the super-resolution capabilities and reduce occasional artifacts like filament duplication in highly complex samples.

Summary updated

Abstract

The spatial resolution of images of living samples obtained by fluorescence microscopes is physically limited due to the diffraction of visible light, which makes the study of entities of size less than the diffraction barrier (around 200 nm in the x-y plane) very challenging. To overcome this limitation, several deconvolution and super-resolution techniques have been proposed. Within the framework of inverse problems, modern approaches in fluorescence microscopy reconstruct a super-resolved image from a temporal stack of frames by carefully designing suitable hand-crafted sparsity-promoting regularisers. Numerically, such approaches are solved by proximal gradient-based iterative schemes. Aiming at obtaining a reconstruction more adapted to sample geometries (e.g. thin filaments), we adopt a plug-and-play denoising approach with convergence guarantees and replace the proximity operator associated with the explicit image regulariser with an image denoiser (i.e. a pre-trained network) which, upon appropriate training, mimics the action of an implicit prior. To account for the independence of the fluctuations between molecules, the model relies on second-order statistics. The denoiser is then trained on covariance images coming from data representing sequences of fluctuating fluorescent molecules with filament structure. The method is evaluated on both simulated and real fluorescence microscopy images, showing its ability to correctly reconstruct filament structures with high values of peak signal-to-noise ratio (PSNR).
BibTeX
@inproceedings{stergiopoulou2023fluctuation,
  author = {Vasiliki Stergiopoulou and Subhadip Mukherjee and Luca Calatroni and Laure Blanc-Féraud},
  title = {{Fluctuation-based deconvolution in fluorescence microscopy using plug-and-play denoisers}},
  booktitle = {SSVM},
  year = {2023},
  doi = {10.1007/978-3-031-31975-4\_38},
}
Figure