A 3D High Resolution Generative Deep-learning Network for Fluorescence Microscopy Image

Hang Zhou, Shiwei Li, Qing Huang, Liu Shijie, Quan Tingwei, Ruiyao Cai, Ali Serol Ertürk, Shaoqun ZengbioRxiv 2019 · DOI

TL;DR

This paper introduces a novel 3D deep learning network, 3D-dualHRGAN, that generates high-resolution fluorescence microscopy images from low-resolution inputs. Crucially, it bypasses the need for precisely registered low- and high-resolution image pairs, a major bottleneck in current methods, by employing a dual-GAN framework with cycle consistency. This advancement promises to make high-resolution volumetric imaging more accessible for biological research.

What Problem Does This Paper Address?

High-resolution imaging is vital for detailed observation of biological specimens, enabling researchers to visualize structures like entire neural networks. However, achieving both high resolution and a large field of view in optical microscopy is fundamentally limited by physics (e.g., Lagrange invariant). While advanced microscopy techniques have made strides, obtaining high-resolution volume images of large biological samples remains challenging. Current deep learning approaches for super-resolution often require massive datasets of precisely registered low-resolution (LR) and high-resolution (HR) image pairs. This registration is particularly difficult for biological samples due to dynamic changes and the inherent differences in feature visibility between LR and HR images. This paper tackles the problem of generating high-resolution volumetric images from low-resolution ones without the stringent requirement of perfect image registration.

Methods

The core of the proposed solution is a 3D deep learning network called 3D-dualHRGAN. This network is built upon a dual generative adversarial network (dual-GAN) framework, incorporating cycle consistency. Instead of directly learning the mapping from LR to HR images, the network uses two GANs: one to generate HR images from LR inputs (HR_G and HR_D) and another to generate LR images from HR inputs (LR_G and HR_G). The cycle consistency ensures that an LR image passed through the LR_G and HR_G generators should ideally return to its original LR form, and similarly for HR images. This approach avoids the need for precise registration.

The network architecture includes:

  • HR Image Generator (HR_G): Composed of an 8-layer short ResBlock and a long-distance ResNet. It utilizes a "linear low-rank convolution" inspired hourglass-shaped ResBlock for efficient feature extraction and a voxel shuffle 3D layer for upsampling.
  • Image Degrading Generator (LR_G): A convolutional neural network designed to map HR to LR images, acting as a downsampling layer.
  • Discriminators (HR_D and LR_D): Convolutional neural networks used to distinguish between real and generated images.

The training employs a WGAN-GP (Wasserstein GAN with Gradient Penalty) method for the GAN components and a traditional Adam algorithm for the cycle-consistent networks. Four loss functions are combined to train the network: adversarial losses for both HR and LR discriminators, and cycle-consistency losses (L1 distance) for both generators.

Datasets

The authors used neuronal images from an adult Thy1-GFPM mouse, where GFP is expressed in neurons. The animal was cleared using the vDISCO pipeline to enable deep imaging. LR images were acquired using a Zeiss Light-sheet Z.1 microscope (5x/0.16NA), and HR images were acquired using a LaVision BioTec Light-sheet Ultramicroscope II (20x/1.0 NA).

For training, the LR and HR volume images were divided into sub-blocks. The sizes of these sub-blocks were set to 32x32x32 for LR and 96x96x96 for HR to manage GPU resource constraints, representing a 3x upsampling factor in each dimension.

Key Insights and Ideas

  • Registration-Free Super-Resolution: The most significant insight is the successful elimination of the need for precise LR-HR image registration. This is a major hurdle in applying deep learning to volumetric biological imaging, and the dual-GAN with cycle consistency effectively overcomes it.
  • Cycle Consistency for Faithfulness: The cycle-consistent network ensures that the generated HR images remain faithful to their corresponding LR inputs, preventing the introduction of unrealistic artifacts and preserving structural integrity.
  • Model-Free Approach: The method is "model-free," meaning it doesn't rely on a specific physical model of the imaging process, making it adaptable to different microscopy modalities.
  • Efficient Architecture: The use of hourglass-shaped ResBlocks (linear low-rank convolution) and voxel shuffling contributes to an efficient architecture capable of achieving good high-resolution prediction with fewer parameters.

Key Results

The 3D-dualHRGAN successfully recovered 20x/1.0 NA resolution volume images from 5x/0.16 NA volume images. This represents a 27:1 voxel ratio increase (3x in each dimension). Qualitative results showed substantial improvement in spatial resolution, with neurites becoming thinner and previously touching cells appearing apart. Weak signals in LR images were also successfully recovered in the predicted HR images. Quantitative analysis of neurite peak widths showed that the predicted HR images had peak widths closer to the ground truth HR images than the original LR images, indicating a significant PSF reduction.

Limitations and Future Work

  • Computational Cost: While the training dataset was downsampled to manage GPU resources, the inherent 27:1 voxel ratio increase means that processing large LR volumes into HR volumes still requires significant computational power.
  • Registration Error: The paper mentions that even with manual feature point pairing, a coarse registration with dozens of voxels of error was achieved. While the network bypasses the need for precise registration, the initial coarse alignment might still influence performance.
  • Future Work: The authors suggest that the method's model-free nature makes it suitable for application to other biological volume imaging systems, implying a broad applicability and potential for further development across different modalities.

Summary updated

Abstract

Abstract Deep learning technology enables us acquire high resolution image from low resolution image in biological imaging free from sophisticated optical hardware. However, current methods require a huge number of the precisely registered low-resolution (LR) and high-resolution (HR) volume image pairs. This requirement is challengeable for biological volume imaging. Here, we proposed 3D deep learning network based on dual generative adversarial network (dual-GAN) framework for recovering HR volume images from LR volume images. Our network avoids learning the direct mappings from the LR and HR volume image pairs, which need precisely image registration process. And the cycle consistent network makes the predicted HR volume image faithful to its corresponding LR volume image. The proposed method achieves the recovery of 20x/1.0 NA volume images from 5x/0.16 NA volume images collected by light-sheet microscopy. In essence our method is suitable for the other imaging modalities.
BibTeX
@misc{zhou2019d,
  author = {Hang Zhou and Shiwei Li and Qing Huang and Liu Shijie and Quan Tingwei and Ruiyao Cai and Ali Serol Ertürk and Shaoqun Zeng},
  title = {{A 3D High Resolution Generative Deep-learning Network for Fluorescence Microscopy Image}},
  year = {2019},
  doi = {10.1101/743179},
  howpublished = {bioRxiv},
}
Figure