Towards Annotation-free Instance Segmentation and Tracking with Adversarial Simulations

Quan Liu, Isabella M. Gaeta, Mengyang Zhao, Ruining Deng, Aadarsh Jha, Bryan A. Millis, Anita Mahadevan-Jansen, Matthew J. Tyska, Yuankai Huo

TL;DR

This paper introduces ASIST, an annotation-free method for instance segmentation and tracking in microscopy videos. By combining adversarial simulations with single-stage pixel-embedding deep learning, ASIST significantly reduces the need for costly manual annotations, achieving comparable or even superior performance to fully supervised methods on both cellular and subcellular imaging tasks.

What Problem Does This Paper Address?

Analyzing biological processes often requires precisely identifying and tracking individual cells and their components within microscopy videos. Traditional methods involve a two-step process: segmenting objects in each frame and then linking them across frames. While recent deep learning approaches have unified this into a single "pixel-embedding" stage, they heavily rely on extensive, pixel-level annotations that are both spatially and temporally consistent. This annotation process is particularly challenging and time-consuming in microscopy due to dense, overlapping objects and dynamic behaviors like cell division. The paper aims to overcome this bottleneck by developing an annotation-free approach.

Methods

The ASIST method is built upon three core stages:

  1. Unsupervised Image-Annotation Synthesis: This stage uses a CycleGAN architecture to learn mappings between real microscope images and their corresponding annotations. Crucially, it leverages simplified representations of object shapes (sticks for microvilli, circles for HeLa cells) to generate synthetic annotations. This bypasses the need for manual annotation of real images for training.
  2. Video Synthesis: The trained CycleGAN's "annotation-to-image" generator is extended to create synthetic videos. Simulated annotation videos are generated with variations in shape, appearance, and dynamics (e.g., translation, rotation, appearing/disappearing, mitosis for HeLa cells). These simulated annotations are then used to generate corresponding synthetic microscope video frames. For HeLa cells, an additional "annotation refinement" step is introduced, involving binary mask generation and non-rigid registration (using ANTs) to better match the complex shapes of these cells. An "annotation cleaning" step is also employed to ensure consistency between generated annotations and binary masks.
  3. Pixel-Embedding Based Instance Segmentation and Tracking: A recurrent stacked hourglass network (RSHN) is trained on the synthetically generated videos and annotations. This network learns pixel-wise embeddings that are consistent for pixels belonging to the same object and distinct for pixels from different objects. The Faster Mean-shift algorithm is then used to cluster these embeddings for instance segmentation and tracking.

Datasets

  • Microvilli Videos: Real microscopy videos of microvilli were used for evaluation. A subset was manually annotated for a small number of frames to serve as a "golden standard" for comparison.
  • HeLa Cell Videos: The N2DL-HeLa dataset from the ISBI Cell Tracking Challenge was used. This dataset provides complete manual annotations for training and testing.

The paper also introduces synthetically generated datasets for training, derived from the adversarial simulation process, for both microvilli and HeLa cells.

Key Insights and Ideas

  • Adversarial Simulation as a Data Augmentation Tool: The paper demonstrates that adversarial simulations, inspired by their success in autonomous driving, can effectively generate realistic synthetic data for microscopy tasks, significantly reducing reliance on manual annotations.
  • Simplified Shape Representations for Annotation Synthesis: Using basic geometric shapes (sticks, circles) as a starting point for annotation synthesis, rather than trying to perfectly replicate complex real annotations, proves to be a viable strategy. This simplifies the generation process while still allowing for effective learning.
  • Annotation Refinement for Complex Cell Shapes: The introduction of an annotation refinement pipeline, including binary mask generation and non-rigid registration, is crucial for capturing the more complex and variable shapes of HeLa cells, improving performance beyond simple circle-based simulations.
  • Annotation Cleaning for Consistency: The annotation cleaning step is highlighted as a key factor in improving performance by ensuring better alignment between the generated synthetic images and their corresponding annotations.

Key Results

  • Microvilli Videos: The ASIST method achieved superior performance compared to fully supervised methods on microvilli videos, with the "Microvilli-20" experiment (using 20 simulated videos) yielding the best results without manual annotations.
  • HeLa Cell Videos: ASIST achieved comparable performance to fully supervised methods on HeLa cell videos. The "HeLa-AD+AC+HW" configuration (with annotation deformation and cleaning, and half window size) showed the best results among the ASIST variants. The performance was reported to be 5% to 9% lower than the manual annotation baseline, which is considered a significant achievement given the annotation-free nature.

Limitations and Future Work

  • Homogeneous Shapes and Appearances: The authors acknowledge that both microvilli and HeLa cells exhibit relatively homogeneous shape and appearance variations. Future work could explore more complex cell lines with greater heterogeneity.
  • Shape Auto-encoders: For more complicated cell shapes, the paper suggests the potential use of shape auto-encoders.
  • Generalizability: The long-term goal is to develop more general and comprehensive algorithms applicable to a wider range of microscope videos, providing new analytical tools for biologists.

Summary updated

Abstract

Background: The quantitative analysis of microscope videos often requires instance segmentation and tracking of cellular and subcellular objects. The traditional method consists of two stages: (1) performing instance object segmentation of each frame, and (2) associating objects frame-by-frame. Recently, pixel-embedding-based deep learning approaches these two steps simultaneously as a single stage holistic solution. In computer vision, annotated training data with consistent segmentation and tracking is resource intensive, the severity of which is multiplied in microscopy imaging due to (1) dense objects (e.g., overlapping or touching), and (2) high dynamics (e.g., irregular motion and mitosis). Adversarial simulations have provided successful solutions to alleviate the lack of such annotations in dynamics scenes in computer vision, such as using simulated environments (e.g., computer games) to train real-world self-driving systems. Methods: In this paper, we propose an annotation-free synthetic instance segmentation and tracking (ASIST) method with adversarial simulation and single-stage pixel-embedding based learning. Contribution: The contribution of this paper is three-fold: (1) the proposed method aggregates adversarial simulations and single-stage pixel-embedding based deep learning; (2) the method is assessed with both the cellular (i.e., HeLa cells) and subcellular (i.e., microvilli) objects; and (3) to the best of our knowledge, this is the first study to explore annotation-free instance segmentation and tracking study for microscope videos. Results: The ASIST method achieved an important step forward, when compared with fully supervised approaches: ASIST shows 7% to 11% higher segmentation, detection and tracking performance on microvilli relative to fully supervised methods, and comparable performance on Hela cell videos.
BibTeX
@article{liu2021towards,
  author = {Quan Liu and Isabella M. Gaeta and Mengyang Zhao and Ruining Deng and Aadarsh Jha and Bryan A. Millis and Anita Mahadevan-Jansen and Matthew J. Tyska and Yuankai Huo},
  title = {{Towards Annotation-free Instance Segmentation and Tracking with Adversarial Simulations}},
  journal = {Computers in Biology and Medicine},
  year = {2021},
  url = {https://openreview.net/forum?id=tHL8uKzOKd_},
}
Figure