MMV_Im2Im: an open-source microscopy machine vision toolbox for image-to-image transformation

Justin Sonneck, Yu Zhou, Jianxu Chen

TL;DR

This paper introduces MMV_Im2Im, a versatile, open-source Python toolbox designed to simplify image-to-image transformations in bioimaging. By leveraging state-of-the-art machine learning engineering practices, it empowers researchers to tackle diverse tasks like segmentation, restoration, and generation without getting bogged down in complex coding, accelerating discovery in biomedical imaging.

What Problem Does This Paper Address?

Deep learning has revolutionized computer vision, but applying these powerful techniques to biomedical imaging, especially microscopy, presents unique challenges. Researchers often face fragmented codebases, difficulty in adapting existing models to new problems, and the steep learning curve associated with advanced machine learning engineering. This leads to duplicated effort, slow progress, and a barrier for experimental biologists who want to leverage AI. The paper addresses this by providing a unified, flexible, and user-friendly framework that democratizes advanced image-to-image transformation for the bioimaging community.

Methods

The core of MMV_Im2Im is built upon PyTorch Lightning, a framework that streamlines PyTorch development by abstracting away much of the boilerplate code for training, validation, and testing. This allows researchers to focus on the model architecture and experimental design rather than low-level engineering. The toolbox is designed with a generic image-to-image transformation framework, enabling it to handle a wide array of tasks including:

  • Supervised Learning: Semantic segmentation, instance segmentation, image restoration, and imaging modality transformation.
  • Unsupervised Learning: Semantic segmentation and image generation.
  • Label-free Prediction: Predicting fluorescent images from brightfield images.

Key engineering aspects include:

  • Modular Design: Facilitates easy extension and customization.
  • Human-Readable Configuration: Uses YAML files for straightforward parameter tuning.
  • Efficient Data Handling: Leverages libraries like aicsimageio and MONAI for handling large, multi-dimensional microscopy datasets (up to 5D) and optimizing training efficiency.
  • State-of-the-Art ML Engineering: Incorporates features like automatic batch size determination, distributed training, and various optimizers.

Datasets

The paper demonstrates MMV_Im2Im's capabilities across numerous publicly available datasets, often sourced from previous publications and challenges. Notable datasets and their applications include:

  • HeLa "Kyoto" cells dataset: Used for 2D label-free prediction of nuclear structure.
  • hiPSC single-cell image dataset (Allen Cell Quilt Bucket): A primary source for 3D label-free prediction, unsupervised semantic segmentation, synthetic image generation, and instance segmentation. This dataset is characterized by its large, high-dimensional, multichannel 3D images.
  • 2015 Gland Segmentation challenge dataset: Used for 2D semantic segmentation of glands in H&E stained tissue images.
  • C. elegans dataset: Used for 2D instance segmentation.
  • Planaria and Tribolium embryos datasets: Used for image denoising.
  • Confocal and STED image datasets: Used for imaging modality transformation.
  • Multiplex imaging datasets: Used for staining transformation.

The paper emphasizes that the scripts for downloading and reorganizing these datasets are included in the repository, promoting reproducibility.

Key Insights and Ideas

  • Unified Framework for Bioimaging AI: The core idea is that many bioimaging tasks can be framed as image-to-image transformations, and a single, well-engineered toolbox can serve as a powerful starting point for a wide range of applications.
  • Democratizing Advanced ML: By abstracting away complex engineering details through PyTorch Lightning and a modular design, MMV_Im2Im significantly lowers the barrier to entry for researchers who may not have extensive ML backgrounds.
  • Handling High-Dimensional Data: The toolbox is specifically designed to efficiently handle the large, multi-dimensional (up to 5D) datasets common in microscopy, a significant challenge for many general computer vision tools.
  • Flexibility through Configuration: The use of human-readable YAML files for configuration allows for easy customization and experimentation with different model architectures, training strategies, and data augmentations.
  • Open Science and Community: The emphasis on an open-source release with comprehensive documentation and tutorials aims to foster a collaborative community, encouraging contributions and further development.

Key Results

MMV_Im2Im was demonstrated across over 10 different biomedical problems with more than 30 experiments. While the paper focuses on showcasing the toolbox's versatility rather than optimizing performance for each specific task, several key findings emerged:

  • Label-free prediction: Achieved visually compelling results for predicting nuclear structures from brightfield images, with models like fnet showing slightly more accurate predictions than basic UNet in 2D. In 3D, center normalization was found to be more robust.
  • Semantic segmentation: Achieved competitive F1-scores (e.g., 0.904 ± 0.060 for glands) on benchmark datasets.
  • Instance segmentation: Demonstrated the ability to handle complex 3D instance segmentation, even with anisotropic data, and showed improved training strategies over existing methods.
  • Image denoising: Showed significant improvements in Pearson correlation for low SNR images (e.g., from 0.065 ± 0.057 to 0.392 ± 0.065 for flatworm data).
  • Imaging modality transformation: Achieved reasonable Pearson correlations (e.g., 0.786 ± 0.020 for microtubule transformation) and improved data quality visually.
  • Synthetic image generation: Produced realistic synthetic microscopy images from binary masks, with supervised methods generally yielding more realistic results than unsupervised ones.

Limitations and Future Work

  • Performance Optimization: The paper explicitly states that the experiments were designed to demonstrate versatility, not to achieve state-of-the-art performance on every task. Further hyperparameter tuning would be necessary for optimal results.
  • Temporal Data: Current frameworks do not explicitly handle temporal information; images are treated independently. Future work aims to incorporate recurrent neural networks (CRNNs) for time-series data.
  • Image Registration: While MMV_Im2Im can perform transformations, image registration (aligning images spatially) is considered a separate problem and is not directly addressed.
  • Unsupervised Denoising & Imaginaire: Support for unsupervised denoising and the Imaginaire framework for more advanced unsupervised learning is planned.
  • Automated Configuration & Active Learning: The development of auxiliary packages, MMV_Im2Im_Auto (for automated configuration) and MMV_Im2Im_Active (for active learning with limited data), is a key future direction.
  • Napari Plugins: Wrapping the toolbox into napari plugins is planned to provide a no-code graphical interface for broader accessibility.
  • Detection Frameworks: While models like Mask-RCNN are powerful, they are considered more suited for detection tasks than image-to-image transformation, suggesting future work on a dedicated detection package (MMV_NDet).

Summary updated

Abstract

Over the past decade, deep learning (DL) research in computer vision has been growing rapidly, with many advances in DL-based image analysis methods for biomedical problems. In this work, we introduce MMV_Im2Im, a new open-source Python package for image-to-image transformation in bioimaging applications. MMV_Im2Im is designed with a generic image-to-image transformation framework that can be used for a wide range of tasks, including semantic segmentation, instance segmentation, image restoration, image generation, and so on. Our implementation takes advantage of state-of-the-art machine learning engineering techniques, allowing researchers to focus on their research without worrying about engineering details. We demonstrate the effectiveness of MMV_Im2Im on more than 10 different biomedical problems, showcasing its general potentials and applicabilities. For computational biomedical researchers, MMV_Im2Im provides a starting point for developing new biomedical image analysis or machine learning algorithms, where they can either reuse the code in this package or fork and extend this package to facilitate the development of new methods. Experimental biomedical researchers can benefit from this work by gaining a comprehensive view of the image-to-image transformation concept through diversified examples and use cases. We hope this work can give the community inspirations on how DL-based image-to-image transformation can be integrated into the assay development process, enabling new biomedical studies that cannot be done only with traditional experimental assays. To help researchers get started, we have provided source code, documentation, and tutorials for MMV_Im2Im at [https://github.com/MMV-Lab/mmv_im2im] under MIT license.
BibTeX
@inproceedings{sonneck2024i,
  author = {Justin Sonneck and Yu Zhou and Jianxu Chen},
  title = {{<i>MMV\_Im2Im</i>: an open-source microscopy machine vision toolbox for image-to-image transformation}},
  booktitle = {GigaScience},
  year = {2024},
  doi = {10.1093/gigascience/giad120},
}
Figure