Deep Learning-Based Point-Scanning Super-Resolution Imaging

Linjing Fang, Fred Monroe, Sammy Weiser Novak, Lyndsey M. Kirk, Cara R. Schiavon, Seungyoon B. Yu, Tong Zhang, Melissa Wu, Kyle Kastner, Yoshiyuki Kubota, Zhao Zhang, Gülçin Pekkurnaz, John M. Mendenhall, Kristen M. Harris, Jeremy Howard, Uri ManorbioRxiv 2019 · DOI

TL;DR

This paper introduces Point-Scanning Super-Resolution (PSSR), a deep learning approach that significantly enhances the resolution and speed of electron and confocal microscopy. By training models on "crappified" high-resolution images, PSSR can restore undersampled images to near-ground truth quality, enabling faster imaging with reduced sample damage and achieving resolutions previously unattainable.

What Problem Does This Paper Address?

Traditional point-scanning microscopy techniques, like scanning electron microscopy (SEM) and laser scanning confocal microscopy (LSM), face a fundamental trade-off: improving image resolution requires acquiring more pixels, which in turn increases imaging time and the risk of sample damage due to prolonged exposure to electron beams or lasers. This "eternal triangle of compromise" limits the ability to simultaneously achieve high resolution, fast acquisition speeds, and minimal sample degradation. For instance, achieving the desired 2 nm resolution in SEM often leads to excessive sample damage, while faster imaging sacrifices crucial detail. Similarly, in fluorescence microscopy, high-resolution Airyscan imaging is slow and can cause phototoxicity, while faster confocal imaging lacks the necessary resolution.

Methods

The core of this work is the development of a deep learning model, termed Point-Scanning Super-Resolution (PSSR), based on a ResNet-U-Net architecture. The authors employ a "crappification" process to generate semi-synthetic training data. This involves taking high-resolution (HR) images and synthetically degrading them to simulate low-resolution (LR) images by applying downsampling, noise (salt-and-pepper, Gaussian), and other distortions. These paired HR and simulated LR images are then used to train the PSSR model. For time-lapse fluorescence imaging, a multi-frame PSSR (PSSR-MF) model was developed, leveraging the temporal continuity of video data by training on five consecutive frames. The training process incorporated advanced techniques like progressive resizing and discriminative learning rates for EM data, and cyclic learning rates and momentum for optimization.

Datasets

The study utilized several datasets for training and evaluation:

  • EM Training Data: ~130 GB of 2 nm pixel transmission-mode scanning electron microscope (tSEM) images of 40 nm ultrathin sections from the hippocampus of a Long Evans male rat.
  • EM Testing Data:
    • Semi-synthetic testing pairs generated from tSEM data.
  • Real-world LR and HR image pairs acquired from mouse brain sections (ZEISS Sigma-VP Gatan Serial Blockface SEM), fly sections (ZEISS/FEI focused ion beam-SEM), mouse sections (ZEISS/FEI FIB-SEM), and rat sections (Hitachi Regulus serial section EM).
  • Fluorescence Training Data: Semi-synthetic LR training data generated from HR Airyscan images of U2OS cells stained with MitoTracker Deep Red or CMXRos Red.
  • Fluorescence Testing Data:
    • Semi-synthetic testing pairs from Airyscan HR movies of U2OS cells.
  • Real-world LR and HR images from U2OS cells and primary hippocampal neurons expressing mito-dsRed.

Key Insights and Ideas

  • "Crappification" as a powerful training strategy: The ability to synthetically degrade high-quality images to mimic real-world low-quality data is a key enabler for training robust PSSR models, significantly reducing the need for perfectly aligned real-world LR/HR pairs.
  • Generalizability across modalities and labs: Remarkably, an EM PSSR model trained on one type of SEM data could effectively restore images acquired with different optics, detectors, samples, and even from different laboratories, highlighting the robustness of the approach.
  • Multi-frame PSSR for dynamic events: For live-cell imaging, using multiple frames in the PSSR model (PSSR-MF) significantly improved the restoration of dynamic events like mitochondrial fission and trafficking, capturing details lost in single-frame or standard confocal imaging.
  • Reduced phototoxicity and sample damage: By enabling faster imaging at lower laser powers, PSSR directly addresses the issue of photobleaching and phototoxicity in fluorescence microscopy, allowing for longer and healthier live-cell imaging.

Key Results

  • EM Imaging:
    • Achieved previously unattainable 2 nm resolution images with a serial block face scanning electron microscope.
  • Demonstrated effective restoration of 8 nm SBFSEM datasets to 2 nm resolution.
  • Showed successful restoration of images from various EM modalities (SBFSEM, FIB-SEM, ssEM) and sample types (mouse, rat, fly).
  • PSSR processing improved manual segmentation of 3DEM datasets.
  • Enabled 25x super-sampling for FIB-SEM imaging, increasing lateral resolution and speed.
  • Fluorescence Imaging:
    • Achieved Airyscan-equivalent spatial resolution and SNR with ~100x lower laser dose and 16x higher frame rates compared to high-resolution acquisitions.
  • Significantly reduced photobleaching and phototoxicity in live-cell imaging of mitochondria.
  • Enabled the detection of mitochondrial fission events that were not visible in standard confocal images.
  • Facilitated the observation of mitochondrial motility with improved temporal resolution.

Limitations and Future Work

  • Accuracy Dependence: The authors acknowledge that any deep learning output is a prediction and never 100% accurate. The sufficiency of accuracy depends on the specific application and error tolerance. For EM, while vesicle detection accuracy was comparable to human experts, other structures might not be restored with the same fidelity.
  • Training Data Similarity: The effectiveness of the PSSR model is still somewhat limited by the similarity between training and testing data. Fluorescence data, being sparser, may require more specific models for different sample types.
  • Loss Function and Model Sophistication: The study primarily used Mean Squared Error (MSE) loss. The authors suggest that more sophisticated loss functions could further improve resolution and SNR.
  • Generalization: While the EM model showed good generalizability, the authors suggest that future work could focus on developing more generalized, unsupervised, or self-supervised denoising approaches for broader applicability across different imaging systems.
  • Transfer Learning and Feature Loss: Employing transfer learning and feature loss could be practical strategies for optimizing models for specific datasets or sample types.
  • Acquisition Scheme Optimization: Future work could involve acquiring a limited number of "ground truth" HR images for fine-tuning models, either before or after low-resolution experimental data acquisition.

Summary updated

Abstract

Point scanning imaging systems (e.g. scanning electron or laser scanning confocal microscopes) are perhaps the most widely used tools for high resolution cellular and tissue imaging. Like all other imaging modalities, the resolution, speed, sample preservation, and signal-to-noise ratio (SNR) of point scanning systems are difficult to optimize simultaneously. In particular, point scanning systems are uniquely constrained by an inverse relationship between imaging speed and pixel resolution. Here we show these limitations can be mitigated via the use of deep learning-based super-sampling of undersampled images acquired on a point-scanning system, which we termed point-scanning super-resolution (PSSR) imaging. Oversampled, high SNR ground truth images acquired on scanning electron or Airyscan laser scanning confocal microscopes were "crappified" to generate semi-synthetic training data for PSSR models that were then used to restore real-world undersampled images. Remarkably, our EM PSSR model could restore undersampled images acquired with different optics, detectors, samples, or sample preparation methods in other labs. PSSR enabled previously unattainable 2 nm resolution images with our serial block face scanning electron microscope system. For fluorescence, we show that undersampled confocal images combined with a multiframe PSSR model trained on Airyscan timelapses facilitates Airyscan-equivalent spatial resolution and SNR with ~100x lower laser dose and 16x higher frame rates than corresponding high-resolution acquisitions. In conclusion, PSSR facilitates point-scanning image acquisition with otherwise unattainable resolution, speed, and sensitivity.
BibTeX
@misc{fang2019deep,
  author = {Linjing Fang and Fred Monroe and Sammy Weiser Novak and Lyndsey M. Kirk and Cara R. Schiavon and Seungyoon B. Yu and Tong Zhang and Melissa Wu and Kyle Kastner and Yoshiyuki Kubota and Zhao Zhang and Gülçin Pekkurnaz and John M. Mendenhall and Kristen M. Harris and Jeremy Howard and Uri Manor},
  title = {{Deep Learning-Based Point-Scanning Super-Resolution Imaging}},
  year = {2019},
  doi = {10.1101/740548},
  howpublished = {bioRxiv},
}
Figure