MARS-Net: Deep learning-based segmentation pipeline for profiling cellular morphodynamics from multiple types of live cell microscopy

Junbong Jang, Chuangqi Wang, Xitong Zhang, Hee June Choi, Xiang Pan, Bolun Lin, Yudong Yu, C. K. Whittle, Madison Ryan, Yenyu Chen, Kwonmoo LeebioRxiv 2017 · DOI

TL;DR

This paper introduces MARS-Net, a novel deep learning framework that leverages data from multiple microscopy types to achieve highly accurate and robust segmentation of live cell boundaries. This advancement is crucial for precisely quantifying cellular morphodynamics, a key process in various biological phenomena.

What Problem Does This Paper Address?

Understanding cellular morphodynamics – the dynamic changes in cell shape and movement – is vital for fields like cancer research and immunology. This understanding relies heavily on accurately tracking cell boundaries over time in live cell imaging. However, live cell imaging presents significant challenges:

  • Image Quality: Live cell images are often noisy, low-contrast, and suffer from photobleaching, making precise edge detection difficult.
  • Artifacts: Different microscopy techniques introduce unique artifacts. For instance, phase contrast microscopy produces halos, while fluorescence microscopy can lead to phototoxicity and low signal.
  • Segmentation Accuracy: Existing segmentation methods, including standard deep learning models like U-Net, struggle to consistently and accurately identify cell edges across diverse imaging conditions and microscopy types. This leads to errors in downstream analyses like velocity estimation.

The authors aim to overcome these limitations by developing a more robust and generalizable segmentation model.

Methods

The core of MARS-Net is a deep neural network architecture based on the U-Net, a popular choice for image segmentation. Here's what's novel:

  • VGG19 Encoder with Transfer Learning: Instead of a standard U-Net encoder, they use a VGG19 encoder pre-trained on ImageNet. This transfer learning approach allows the model to leverage general image features learned from a massive dataset, improving its ability to generalize to new, unseen data.
  • Multiple Microscopy-Type Training: The key innovation is training the model on a combined dataset from phase contrast, spinning disk confocal (SDC), and total internal reflection fluorescence (TIRF) microscopy. The hypothesis is that learning from diverse imaging modalities will lead to more robust feature extraction for cell edge localization.
  • Dropout Layers: Dropout layers are incorporated into the VGG19D-U-Net architecture as a regularization technique to prevent overfitting and improve performance.
  • Labeling Tool: A custom semi-automatic labeling tool was developed to streamline the creation of ground truth masks, reducing human labor and improving consistency.

Datasets

The study utilized live cell movies of migrating PtK1 cells acquired using three different microscopy techniques:

  • Phase Contrast Microscopy: Five label-free movies.
  • Spinning Disk Confocal (SDC) Microscopy: Five dual-color movies (GFP-mDia1).
  • Total Internal Reflection Fluorescence (TIRF) Microscopy: Six movies (paxillin-HaloTag-TMR).

Ground truth masks were generated using their custom labeling tool and, in some cases, standard image processing techniques (e.g., thresholding actin images for SDC). The dataset was structured for leave-one-movie-out cross-validation to rigorously assess generalization.

Key Insights and Ideas

  • Cross-Modal Learning is Key: Training a single model on data from multiple microscopy types (MARS-Net) significantly improves segmentation accuracy and robustness compared to models trained on single microscopy types, even when using a powerful architecture like VGG19D-U-Net. This suggests that diverse imaging conditions teach the model more generalizable features for cell boundary detection.
  • Transfer Learning Matters: Pre-training the VGG19 encoder on ImageNet provides a substantial performance boost, enabling the model to achieve high accuracy even with limited training data.
  • Dropout Enhances Robustness: Adding dropout layers to the VGG19 encoder further improves segmentation performance without increasing model complexity.
  • MARS-Net Handles Challenging Images: The MARS-Net model demonstrates superior performance in segmenting difficult images, such as those with halo artifacts (phase contrast) or low contrast and noise (SDC/TIRF), where standard U-Net struggles.

Key Results

  • Improved F1 Scores: MARS-Net achieved significantly higher F1 scores across all microscopy types compared to single-microscopy-trained models. For example, on phase contrast data, MARS-Net achieved an F1 score of 0.943, outperforming U-Net.
  • Robustness to Data Scarcity: MARS-Net could achieve higher F1 scores with fewer training frames (e.g., 1 frame) than a standard U-Net trained on many more frames (e.g., 34 frames).
  • Reduced Errors in Morphodynamic Profiling: Protrusion maps generated from MARS-Net segmentation showed fewer errors and less noise compared to those from U-Net, leading to more accurate quantitative profiling of cellular morphodynamics.
  • Effective Feature Learning: Class activation maps revealed that MARS-Net learns features from both inside and outside the cell boundary, allowing it to better handle artifacts like phase contrast halos, whereas single-microscopy models focused more narrowly.

Limitations and Future Work

  • Computational Cost: While MARS-Net offers significant performance gains, the VGG19 encoder, especially with dropout, can increase training time compared to a standard U-Net.
  • Generalizability to Unseen Modalities: While MARS-Net generalizes well across the tested microscopy types, its performance on entirely new, unseen microscopy modalities is not explicitly discussed.
  • Further Optimization: The authors suggest that further exploration of different encoder architectures and regularization techniques could lead to even greater improvements. The paper also highlights the potential for MARS-Net to be applied to other cell types and biological processes.

Summary updated

Abstract

Abstract Quantitative studies of cellular morphodynamics rely on extracting leading-edge velocity time-series based on accurate cell segmentation from live cell imaging. However, live cell imaging has numerous challenging issues about accurate edge localization. Here, we develop a deep learning-based pipeline, termed MARS-Net (Multiple-microscopy- type-based Accurate and Robust Segmentation Network), that utilizes transfer learning and the datasets from multiple types of microscopy to localize cell edges with high accuracy, allowing quantitative profiling of cellular morphodynamics. For effective training with the datasets from multiple types of live cell microscopy, we integrated the pretrained VGG-19 encoder with U-Net decoder and added dropout layers. Using this structure, we were able to train one neural network model that can accurately segment various live cell movies from phase contrast, spinning disk confocal, and total internal reflection fluorescence microscopes. Intriguingly, MARS-Net produced more accurate edge localization than the neural network models trained with single microscopy type datasets, whereas the standard U-Net could not increase the overall accuracy. We expect that MARS-Net can accelerate the studies of cellular morphodynamics by providing accurate segmentation of challenging live cell images.
BibTeX
@misc{jang2017mars,
  author = {Junbong Jang and Chuangqi Wang and Xitong Zhang and Hee June Choi and Xiang Pan and Bolun Lin and Yudong Yu and C. K. Whittle and Madison Ryan and Yenyu Chen and Kwonmoo Lee},
  title = {{MARS-Net: Deep learning-based segmentation pipeline for profiling cellular morphodynamics from multiple types of live cell microscopy}},
  year = {2017},
  doi = {10.1101/191858},
  howpublished = {bioRxiv},
}
Figure