Deep Learning Framework For Mobile Microscopy

Anatasiia Kornilova, Mikhail Salnikov, Olga Novitskaya, Maria Begicheva, Egor Sevriugov, Kirill Shcherbakov, Valeriya Pronina, Dmitry V. Dylov

TL;DR

This paper introduces a deep learning framework designed to overcome the image quality limitations of mobile microscopy, a promising tool for accessible disease diagnostics. By adapting and enhancing existing deep learning models for in-focus/out-of-focus classification, image deblurring, and focus-stacking, the authors create a pipeline that significantly improves the usability of smartphone-based microscopy for high-throughput applications.

What Problem Does This Paper Address?

Mobile microscopy offers a low-cost and portable solution for disease diagnostics and at-home health monitoring. However, its widespread adoption is hampered by the inherently lower image quality compared to professional clinical microscopes. This is due to factors like cheaper optics, sensor limitations, and artifacts such as dust or condensation. While some image enhancement techniques exist, they are often developed for professional microscopes and don't directly translate to the unique challenges of mobile microscopy. Specifically, automating tasks like filtering out blurry images, sharpening fast-scanned images, and creating a complete 3D view from multiple focal planes requires specialized approaches. This paper aims to bridge this gap by developing a tailored deep learning pipeline for these critical image processing tasks in the mobile microscopy context.

Methods

The authors propose a three-pronged deep learning approach:

  1. In-focus/out-of-focus classification: They adapt a multi-label classification CNN, inspired by existing work but trained on actual mobile microscopy data. A key modification is the use of a "rank probability score" instead of standard cross-entropy, which they found to yield better results for their task. They also employ image cropping to focus on relevant areas and use 10 focus levels for classification.
  2. Fast scanning image deblurring: To address the blur introduced by fast scanning, they explore and modify existing deep learning architectures. They start with U-Net and SRCNN but focus on an upgraded version of DeblurGAN (uDeblurGAN). This upgraded model replaces 2D transpose convolutions with resize-convolution layers, which they found helps to reduce checkerboard artifacts and improve image quality. Pre-training the generator on input images is also used to prevent discriminator overfitting.
  3. Focus-stacking (Multi-Focus Image Fusion): For creating a complete 3D view, they adapt the FuseGAN model. Their adaptation involves using a modified Harris Corner Response Measurement for mask generation and exploring different loss function combinations for the generator. They also compare their approach to classical methods like wavelet transforms.

Datasets

The primary dataset used for training and evaluation consists of a z-stack video dataset of 150 specimens acquired using a MEL Science auto-scanning mobile microscope. For the in-focus/out-of-focus classification, they created a dataset with 45 specimens, artificially introducing 20 levels of blur. For evaluation, they used 100 focused and 100 unfocused crops, hand-labeled. For focus-stacking, due to the lack of ground truth, they used images obtained from the IFCNN model as a proxy for ground truth.

Key Insights and Ideas

  • Mobile microscopy has unique challenges: Standard image processing techniques designed for professional microscopes often fail on mobile microscopy data due to artifacts and lower inherent quality.
  • Tailoring DL is crucial: Simply applying existing deep learning models isn't enough; modifications and retraining on mobile microscopy data are essential for optimal performance.
  • Rank probability score for focus classification: This alternative to cross-entropy proved more effective for classifying focus levels in their specific mobile microscopy context.
  • uDeblurGAN's advantage: Replacing transpose convolutions with resize-convolutions in DeblurGAN significantly reduced artifacts and improved deblurring quality.
  • Qualitative vs. Quantitative: For focus-stacking, while quantitative metrics didn't always improve, expert visual evaluation showed a clear preference for their modified FuseGAN approach, highlighting the importance of human perception in image quality assessment.

Key Results

  • In-focus/out-of-focus classification: Their proposed model achieved an accuracy of 95.6 ± 2.8%, outperforming traditional focus measures, SVM, and a pre-trained model for general images.
  • Image Deblurring: While SRCNN showed good PSNR and SSIM, their uDeblurGAN model produced sharper details. Compared to the standard DeblurGAN (sDeblurGAN), uDeblurGAN achieved significantly higher PSNR and SSIM, with sDeblurGAN producing noticeable artifacts.
  • Focus-stacking: An anonymous survey of 28 bioimaging experts indicated that their upgraded FuseGAN model was qualitatively the best for image fusion.

Limitations and Future Work

  • Dataset limitations: Validation on other microscope setups is a scope for future work. The focus-stacking evaluation relied on proxy ground truth (IFCNN) and no-reference quality assessment metrics, as true ground truth was unavailable.
  • Quantitative metrics for focus-stacking: While visually superior, their focus-stacking method did not always improve quantitative metrics like BRISQUE and CNNIQA, suggesting these metrics might not fully capture the perceived quality improvements.
  • End-to-end pipeline: The authors envision a future where these modules are integrated into a comprehensive, end-to-end pipeline for automated mobile microscopy analysis.

Summary updated

Abstract

Mobile microscopy is a promising technology to assist and to accelerate disease diagnostics, with its widespread adoption being hindered by the mediocre quality of acquired images. Although some paired image translation and super-resolution approaches for mobile microscopy have emerged, a set of essential challenges, necessary for automating it in a highthroughput setting, still await to be addressed. The issues like in-focus/out-of-focus classification, fast scanning deblurring, focus-stacking, etc. - all have specific peculiarities when the data are recorded using a mobile device. In this work, we aspire to create a comprehensive pipeline by connecting a set of methods purposely tuned to mobile microscopy: (1) a CNN model for stable in-focus/out-of-focus classification, (2) modified DeblurGAN architecture for image deblurring, (3) FuseGAN model for combining in-focus parts from multiple images to boost the detail. We discuss the limitations of the existing solutions developed for professional clinical microscopes, propose corresponding improvements, and compare to the other state-of-the-art mobile analytics solutions.
BibTeX
@inproceedings{kornilova2021deep,
  author = {Anatasiia Kornilova and Mikhail Salnikov and Olga Novitskaya and Maria Begicheva and Egor Sevriugov and Kirill Shcherbakov and Valeriya Pronina and Dmitry V. Dylov},
  title = {{Deep Learning Framework For Mobile Microscopy}},
  booktitle = {ISBI},
  year = {2021},
  url = {https://openreview.net/forum?id=1SE5CrIDFt},
}
Figure