Weakly Supervised Cell-Instance Segmentation with Two Types of Weak Labels by Single Instance Pasting

Kazuya Nishimura, Ryoma BiseWACV 2023 · OpenReview

Abstract

Cell instance segmentation that recognizes each cell boundary is an important task in cell image analysis. While deep learning-based methods have shown promising performances with a certain amount of training data, most of them require full annotations that show the boundary of each cell. Generating the annotation for cell segmentation is time-consuming and human labor. To reduce the annotation cost, we propose a weakly supervised segmentation method using two types of weak labels (one for cell type and one for nuclei position). Unlike general images, these two labels are easily obtained in phase-contrast images. The intercellular boundary, which is necessary for cell instance segmentation, cannot be directly obtained from these two weak labels, so to generate the boundary information, we propose a single instance pasting based on the copy-and-paste technique. First, we locate single-cell regions by counting cells and store them in a pool. Then, we generate the intercel-lular boundary by pasting the stored single-cell regions to the original image. Finally, we train a boundary estimation network with the generated labels and perform instance segmentation with the network. Our evaluation on a public dataset demonstrated that the proposed method achieves the best performance among the several weakly supervised methods we compared.

Introduction

In biological research, segmenting individual cells in phase-contrast microscopy is crucial for analyzing cell behavior and morphology. However, training deep learning models for this task typically requires thousands of manual annotations where a human has painstakingly traced the boundary of every single cell. This is a massive bottleneck: it’s slow, labor-intensive, and prone to human error.

The core challenge in "weakly supervised" learning for cells is the intercellular boundary. In phase-contrast images, the contrast is low, and when two cells touch, the line between them is often nearly invisible. While a dot can tell a model where a cell is, it doesn't tell the model where one cell ends and the next begins.

Overview of the aim and weak labels used.

Methods

The proposed framework leverages two types of weak labels: image-level labels (identifying the cell type) and point-level labels (marking the center of the nuclei).

Pipeline Overview

  1. Foreground and Detection Training: A network is trained to estimate the general foreground area using Class Activation Maps (CAMs), while another detects cell positions using point labels.
  2. Single Instance Pasting: This innovation exploits the temporal dynamics of cell cultures. Early frames in time-lapse microscopy are often sparse. The authors identify "single-cell regions" by counting how many nuclei points fall into a connected foreground segment. If the count is exactly one, that cell is cropped and stored in a "pool."
  3. Generating Synthetic Boundaries: To teach the model what a boundary looks like, single cells from the pool are "pasted" onto real images. Because the location of the pasted cell is known, the system has a perfect, self-generated ground truth for the new intercellular boundary created by the overlap.
  4. Boundary Estimation: A boundary estimation network ($f_b$) is trained on these synthetic images to predict a distance-transform map, where cell centers have high values and edges have low values.
  5. Final Segmentation: During inference, the predicted boundary map and detected nuclei points are combined using a marker-controlled watershed algorithm to produce final instance masks.

Experimental Results

The method was evaluated on the LIVECell dataset, a large-scale benchmark containing eight different cell types with varying morphologies. Manual annotations were used to simulate weak labels (using bounding box centers as point labels) for fair comparison.

Performance

The proposed method outperformed existing weakly supervised and unsupervised methods across several metrics:

  • Accuracy: On the LIVECell dataset, it achieved an average Pixel-level F1 score ($F1_p$) of 0.836, significantly higher than competitors like Nishimura et al. (0.764) or Chalfoun et al. (0.820).
  • Robustness: The method proved effective across diverse cell morphologies. For the A172 cell type, the weakly supervised performance (0.921 $F1_p$) rivaled fully supervised U-Net models (0.844 $F1_p$).
  • Ablation Study: Results showed that "Single Instance Pasting" was the critical component; without it, the model’s ability to separate touching cells dropped significantly.

Discussion and Conclusion

While the method shows state-of-the-art performance for weakly supervised segmentation, it has limitations. Some cell types (like BT-474) exhibit morphological plasticity, changing shape upon contact. Since the "pasting" method uses isolated single-cell shapes, it may not perfectly capture these changes. Additionally, extremely dense cultures from the first frame might lack sufficient single cells for the instance pool.

Future work could involve integrating mechanisms to "deform" pasted cells to simulate contact-induced shape changes, further improving accuracy for complex morphologies. Overall, this approach significantly reduces the annotation burden for cell instance segmentation by effectively utilizing easily obtainable weak labels.

Summary updated

Abstract

Cell instance segmentation that recognizes each cell boundary is an important task in cell image analysis. While deep learning-based methods have shown promising performances with a certain amount of training data, most of them require full annotations that show the boundary of each cell. Generating the annotation for cell segmentation is time-consuming and human labor. To reduce the annotation cost, we propose a weakly supervised segmentation method using two types of weak labels (one for cell type and one for nuclei position). Unlike general images, these two labels are easily obtained in phase-contrast images. The intercellular boundary, which is necessary for cell instance segmentation, cannot be directly obtained from these two weak labels, so to generate the boundary information, we propose a single instance pasting based on the copy-and-paste technique. First, we locate single-cell regions by counting cells and store them in a pool. Then, we generate the intercel-lular boundary by pasting the stored single-cell regions to the original image. Finally, we train a boundary estimation network with the generated labels and perform instance segmentation with the network. Our evaluation on a public dataset demonstrated that the proposed method achieves the best performance among the several weakly supervised methods we compared.
BibTeX
@inproceedings{nishimura2023weakly,
  author = {Kazuya Nishimura and Ryoma Bise},
  title = {{Weakly Supervised Cell-Instance Segmentation with Two Types of Weak Labels by Single Instance Pasting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=4YRiEBqNA2},
}
Figure