Teaching deep neural networks to localize sources in super-resolution microscopy by combining simulation-based learning and unsupervised learning

Artur Speiser, Srinivas C. Turaga, Jakob H. MackearXiv 2019 · OpenReview

TL;DR

This paper introduces DECODE, a novel deep learning framework that significantly advances single-molecule localization microscopy (SMLM). By leveraging temporal context and a Bayesian inference approach, DECODE achieves state-of-the-art performance in detecting and localizing fluorophores, particularly in challenging high-density scenarios, and provides crucial uncertainty estimates.

What Problem Does This Paper Address?

Super-resolution microscopy techniques like STORM and PALM allow us to visualize biological structures beyond the diffraction limit of light. The core of these methods, Single Molecule Localization Microscopy (SMLM), relies on accurately pinpointing the locations of individual fluorescent molecules. Traditional methods often involve a two-step process: first detecting potential fluorophore signals, then localizing them. While effective in ideal conditions (high signal-to-noise ratio, low molecule density), these methods struggle with dense samples where multiple fluorophores overlap, leading to inaccurate or missed detections. Furthermore, most existing algorithms process each image frame independently, ignoring valuable temporal information from sequential frames that could help distinguish closely spaced or transiently active molecules. This paper aims to overcome these limitations by developing a more robust and accurate localization method.

Methods

The authors introduce DECODE (DEep Context Dependent), a deep neural network architecture designed for simultaneous detection and localization of fluorophores. Key innovations include:

  • Novel Network Architecture: DECODE processes sequences of imaging frames, integrating information across multiple frames using a "temporal context module." This allows it to leverage the fact that a single fluorophore can be active for several consecutive frames.
  • Bayesian Inference and Uncertainty Estimation: The localization problem is framed as a Bayesian inference task. DECODE's loss function is designed to not only predict fluorophore locations but also to provide uncertainty estimates for these predictions. This is a significant departure from previous methods that often only provided point estimates.
  • Simulator Learning (SL): The network is trained using simulated SMLM data, where known fluorophore positions are used to generate realistic noisy images. This "simulator learning" approach is common but relies heavily on the accuracy of the simulation model.
  • Combined Learning (CL): To address potential mismatches between simulated and real data, the authors propose a combined learning approach that integrates simulator learning with variational autoencoder (VAE) learning. This allows the network to learn from real data and refine its understanding of the imaging process.

Datasets

The paper utilizes several datasets for training and evaluation:

  • SMLM 2016 Challenge Datasets: These are publicly available synthetic datasets designed to emulate various experimental conditions (different signal-to-noise ratios, emitter densities, and point spread functions). DECODE was evaluated on all 12 of these datasets.
  • dSTORM Microtubule Datasets: Real experimental data of labeled microtubules imaged at four different emitter densities were used to demonstrate DECODE's ability to achieve accurate reconstructions with shorter imaging times.
  • LLS-PAINT COS-7 Cell Dataset: A large 3D dataset of a COS-7 cell was used to showcase DECODE's applicability to different imaging modalities and its ability to resolve fine structural details.

Key Insights and Ideas

  • Temporal Context is Crucial: Exploiting the temporal correlation of fluorophore activity across multiple frames significantly improves both detection and localization accuracy, especially in challenging dense conditions.
  • Uncertainty is Informative: Providing uncertainty estimates for localizations is not just an add-on; it's a powerful tool for filtering out less reliable detections and improving the overall quality of the reconstructed super-resolution image.
  • Deep Learning for Bayesian Inference: Framing localization as a Bayesian inference problem and using deep learning to approximate the posterior distribution is a promising avenue for solving complex inverse problems in imaging.
  • Robustness Through Combined Learning: The combination of simulator learning and unsupervised learning (like VAEs) makes the localization model more robust to discrepancies between simulated training data and real-world experimental conditions.

Key Results

  • State-of-the-Art Performance: DECODE achieved the best performance on all 12 datasets of the SMLM 2016 challenge, outperforming all other ranked algorithms.
  • Significant Improvement in High-Density Scenarios: DECODE showed a substantial advantage (up to 10x improvement) over existing methods on high-density datasets, where traditional methods struggle.
  • Reduced Imaging Time: For real dSTORM data, DECODE enabled reconstructions with similar quality using 10x fewer imaging frames, demonstrating its efficiency.
  • Improved Resolution: DECODE consistently improved resolution by 10-25 nm over the CSpline algorithm across various imaging densities.
  • Accurate Uncertainty Estimates: DECODE's uncertainty estimates closely matched analytical Cramér-Rao bounds in sparse data and provided more realistic (and often lower) uncertainties in dense data compared to methods that don't account for temporal dynamics.

Limitations and Future Work

  • Training Time: Training a DECODE network can be computationally intensive, taking over 10 hours on a single GPU, especially when optical properties change.
  • Adaptability to New Microscopes: Currently, a new network needs to be trained for each change in microscope optical properties. The authors suggest that training a single, more general network that can adapt to minor variations in PSF or noise could be a future direction.
  • Generalizability of Bayesian Inverse Problems: The paper highlights that the lessons learned in applying DECODE to SMLM are broadly applicable to other challenging Bayesian inverse problems in biology and physics. This suggests future work could extend DECODE's principles to other scientific domains.

Summary updated

Abstract

Single-molecule localization fluorescence microscopy constructs super-resolution images by sequential imaging and computational localization of sparsely activated fluorophores. Accurate and efficient fluorophore localization algorithms are key to the success of this computational microscopy method. We present a novel localization algorithm based on deep learning which significantly improves upon the state of the art. Our contributions are a novel network architecture for simultaneous detection and localization, and new loss function which phrases detection and localization as a Bayesian inference problem, and thus allows the network to provide uncertainty-estimates. In contrast to standard methods which independently process imaging frames, our network architecture uses temporal context from multiple sequentially imaged frames to detect and localize molecules. We demonstrate the power of our method across a variety of datasets, imaging modalities, signal to noise ratios, and fluorophore densities. While existing localization algorithms can achieve optimal localization accuracy at low fluorophore densities, they are confounded by high densities. Our method is the first deep-learning based approach which achieves state-of-the-art on the SMLM2016 challenge. It achieves the best scores on 12 out of 12 data-sets when comparing both detection accuracy and precision, and excels at high densities. Finally, we investigate how unsupervised learning can be used to make the network robust against mismatch between simulated and real data. The lessons learned here are more generally relevant for the training of deep networks to solve challenging Bayesian inverse problems on spatially extended domains in biology and physics.
BibTeX
@misc{speiser2019teaching,
  author = {Artur Speiser and Srinivas C. Turaga and Jakob H. Macke},
  title = {{Teaching deep neural networks to localize sources in super-resolution microscopy by combining simulation-based learning and unsupervised learning}},
  year = {2019},
  eprint = {1907.00770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.00770},
}
Figure