TL;DR
This paper introduces ASIST, an annotation-free method for instance segmentation and tracking in microscopy videos. By combining adversarial simulations with single-stage pixel-embedding deep learning, ASIST significantly reduces the need for costly manual annotations, achieving comparable or even superior performance to fully supervised methods on both cellular and subcellular imaging tasks.
What Problem Does This Paper Address?
Analyzing biological processes often requires precisely identifying and tracking individual cells and their components within microscopy videos. Traditional methods involve a two-step process: segmenting objects in each frame and then linking them across frames. While recent deep learning approaches have unified this into a single "pixel-embedding" stage, they heavily rely on extensive, pixel-level annotations that are both spatially and temporally consistent. This annotation process is particularly challenging and time-consuming in microscopy due to dense, overlapping objects and dynamic behaviors like cell division. The paper aims to overcome this bottleneck by developing an annotation-free approach.
Methods
The ASIST method is built upon three core stages:
- Unsupervised Image-Annotation Synthesis: This stage uses a CycleGAN architecture to learn mappings between real microscope images and their corresponding annotations. Crucially, it leverages simplified representations of object shapes (sticks for microvilli, circles for HeLa cells) to generate synthetic annotations. This bypasses the need for manual annotation of real images for training.
- Video Synthesis: The trained CycleGAN's "annotation-to-image" generator is extended to create synthetic videos. Simulated annotation videos are generated with variations in shape, appearance, and dynamics (e.g., translation, rotation, appearing/disappearing, mitosis for HeLa cells). These simulated annotations are then used to generate corresponding synthetic microscope video frames. For HeLa cells, an additional "annotation refinement" step is introduced, involving binary mask generation and non-rigid registration (using ANTs) to better match the complex shapes of these cells. An "annotation cleaning" step is also employed to ensure consistency between generated annotations and binary masks.
- Pixel-Embedding Based Instance Segmentation and Tracking: A recurrent stacked hourglass network (RSHN) is trained on the synthetically generated videos and annotations. This network learns pixel-wise embeddings that are consistent for pixels belonging to the same object and distinct for pixels from different objects. The Faster Mean-shift algorithm is then used to cluster these embeddings for instance segmentation and tracking.
Datasets
- Microvilli Videos: Real microscopy videos of microvilli were used for evaluation. A subset was manually annotated for a small number of frames to serve as a "golden standard" for comparison.
- HeLa Cell Videos: The N2DL-HeLa dataset from the ISBI Cell Tracking Challenge was used. This dataset provides complete manual annotations for training and testing.
The paper also introduces synthetically generated datasets for training, derived from the adversarial simulation process, for both microvilli and HeLa cells.
Key Insights and Ideas
- Adversarial Simulation as a Data Augmentation Tool: The paper demonstrates that adversarial simulations, inspired by their success in autonomous driving, can effectively generate realistic synthetic data for microscopy tasks, significantly reducing reliance on manual annotations.
- Simplified Shape Representations for Annotation Synthesis: Using basic geometric shapes (sticks, circles) as a starting point for annotation synthesis, rather than trying to perfectly replicate complex real annotations, proves to be a viable strategy. This simplifies the generation process while still allowing for effective learning.
- Annotation Refinement for Complex Cell Shapes: The introduction of an annotation refinement pipeline, including binary mask generation and non-rigid registration, is crucial for capturing the more complex and variable shapes of HeLa cells, improving performance beyond simple circle-based simulations.
- Annotation Cleaning for Consistency: The annotation cleaning step is highlighted as a key factor in improving performance by ensuring better alignment between the generated synthetic images and their corresponding annotations.
Key Results
- Microvilli Videos: The ASIST method achieved superior performance compared to fully supervised methods on microvilli videos, with the "Microvilli-20" experiment (using 20 simulated videos) yielding the best results without manual annotations.
- HeLa Cell Videos: ASIST achieved comparable performance to fully supervised methods on HeLa cell videos. The "HeLa-AD+AC+HW" configuration (with annotation deformation and cleaning, and half window size) showed the best results among the ASIST variants. The performance was reported to be 5% to 9% lower than the manual annotation baseline, which is considered a significant achievement given the annotation-free nature.
Limitations and Future Work
- Homogeneous Shapes and Appearances: The authors acknowledge that both microvilli and HeLa cells exhibit relatively homogeneous shape and appearance variations. Future work could explore more complex cell lines with greater heterogeneity.
- Shape Auto-encoders: For more complicated cell shapes, the paper suggests the potential use of shape auto-encoders.
- Generalizability: The long-term goal is to develop more general and comprehensive algorithms applicable to a wider range of microscope videos, providing new analytical tools for biologists.
Summary updated