TL;DR
This paper introduces a deep learning framework designed to overcome the image quality limitations of mobile microscopy, a promising tool for accessible disease diagnostics. By adapting and enhancing existing deep learning models for in-focus/out-of-focus classification, image deblurring, and focus-stacking, the authors create a pipeline that significantly improves the usability of smartphone-based microscopy for high-throughput applications.
What Problem Does This Paper Address?
Mobile microscopy offers a low-cost and portable solution for disease diagnostics and at-home health monitoring. However, its widespread adoption is hampered by the inherently lower image quality compared to professional clinical microscopes. This is due to factors like cheaper optics, sensor limitations, and artifacts such as dust or condensation. While some image enhancement techniques exist, they are often developed for professional microscopes and don't directly translate to the unique challenges of mobile microscopy. Specifically, automating tasks like filtering out blurry images, sharpening fast-scanned images, and creating a complete 3D view from multiple focal planes requires specialized approaches. This paper aims to bridge this gap by developing a tailored deep learning pipeline for these critical image processing tasks in the mobile microscopy context.
Methods
The authors propose a three-pronged deep learning approach:
- In-focus/out-of-focus classification: They adapt a multi-label classification CNN, inspired by existing work but trained on actual mobile microscopy data. A key modification is the use of a "rank probability score" instead of standard cross-entropy, which they found to yield better results for their task. They also employ image cropping to focus on relevant areas and use 10 focus levels for classification.
- Fast scanning image deblurring: To address the blur introduced by fast scanning, they explore and modify existing deep learning architectures. They start with U-Net and SRCNN but focus on an upgraded version of DeblurGAN (uDeblurGAN). This upgraded model replaces 2D transpose convolutions with resize-convolution layers, which they found helps to reduce checkerboard artifacts and improve image quality. Pre-training the generator on input images is also used to prevent discriminator overfitting.
- Focus-stacking (Multi-Focus Image Fusion): For creating a complete 3D view, they adapt the FuseGAN model. Their adaptation involves using a modified Harris Corner Response Measurement for mask generation and exploring different loss function combinations for the generator. They also compare their approach to classical methods like wavelet transforms.
Datasets
The primary dataset used for training and evaluation consists of a z-stack video dataset of 150 specimens acquired using a MEL Science auto-scanning mobile microscope. For the in-focus/out-of-focus classification, they created a dataset with 45 specimens, artificially introducing 20 levels of blur. For evaluation, they used 100 focused and 100 unfocused crops, hand-labeled. For focus-stacking, due to the lack of ground truth, they used images obtained from the IFCNN model as a proxy for ground truth.
Key Insights and Ideas
- Mobile microscopy has unique challenges: Standard image processing techniques designed for professional microscopes often fail on mobile microscopy data due to artifacts and lower inherent quality.
- Tailoring DL is crucial: Simply applying existing deep learning models isn't enough; modifications and retraining on mobile microscopy data are essential for optimal performance.
- Rank probability score for focus classification: This alternative to cross-entropy proved more effective for classifying focus levels in their specific mobile microscopy context.
- uDeblurGAN's advantage: Replacing transpose convolutions with resize-convolutions in DeblurGAN significantly reduced artifacts and improved deblurring quality.
- Qualitative vs. Quantitative: For focus-stacking, while quantitative metrics didn't always improve, expert visual evaluation showed a clear preference for their modified FuseGAN approach, highlighting the importance of human perception in image quality assessment.
Key Results
- In-focus/out-of-focus classification: Their proposed model achieved an accuracy of 95.6 ± 2.8%, outperforming traditional focus measures, SVM, and a pre-trained model for general images.
- Image Deblurring: While SRCNN showed good PSNR and SSIM, their uDeblurGAN model produced sharper details. Compared to the standard DeblurGAN (sDeblurGAN), uDeblurGAN achieved significantly higher PSNR and SSIM, with sDeblurGAN producing noticeable artifacts.
- Focus-stacking: An anonymous survey of 28 bioimaging experts indicated that their upgraded FuseGAN model was qualitatively the best for image fusion.
Limitations and Future Work
- Dataset limitations: Validation on other microscope setups is a scope for future work. The focus-stacking evaluation relied on proxy ground truth (IFCNN) and no-reference quality assessment metrics, as true ground truth was unavailable.
- Quantitative metrics for focus-stacking: While visually superior, their focus-stacking method did not always improve quantitative metrics like BRISQUE and CNNIQA, suggesting these metrics might not fully capture the perceived quality improvements.
- End-to-end pipeline: The authors envision a future where these modules are integrated into a comprehensive, end-to-end pipeline for automated mobile microscopy analysis.
Summary updated