CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation

Shengcong Chen, Changxing Ding, Minfeng Liu, Dacheng TaoIEEE TIP 2021 · DOI · OpenReview

TL;DR

This paper introduces CPP-Net, a novel deep learning model for nucleus segmentation in microscopy images. By incorporating contextual information from multiple points around a nucleus's centroid and a new shape-aware loss function, CPP-Net significantly improves the accuracy and robustness of nucleus boundary prediction, especially for crowded and overlapping cells.

What Problem Does This Paper Address?

Accurately segmenting nuclei in microscopy images is crucial for various biological analyses, from counting cells to understanding their phenotypes. However, this task is notoriously difficult due to nuclei often being densely packed and having fuzzy, indistinct boundaries. Traditional methods struggle with overlapping nuclei, and even recent polygon-based approaches that represent nuclei as polygons have limitations. These methods typically rely on features from a single centroid pixel to predict the distance to the nucleus boundary. This single point lacks sufficient contextual information, especially for larger nuclei where the centroid is far from the boundary, leading to degraded segmentation accuracy. The paper aims to overcome this by leveraging richer contextual information and enforcing shape constraints.

Methods

CPP-Net builds upon the StarDist framework but introduces several key innovations:

  • Context Enhancement Module (CEM): Instead of using only the centroid pixel, CEM samples a set of points between the centroid and the predicted boundary along multiple directions. This allows the model to gather more contextual information from pixels closer to the actual boundary, leading to more accurate distance predictions.
  • Confidence-based Weighting Module (CWM): The confidence of the predictions from the sampled points can vary. CWM adaptively assigns weights to these sampled points based on their prediction confidence, allowing the model to fuse this information more robustly and mitigate the impact of potentially noisy samples.
  • Shape-Aware Perceptual (SAP) Loss: To better constrain the predicted nucleus shapes, the authors introduce a novel SAP loss. This loss is inspired by perceptual losses used in image style transfer. It involves training an auxiliary encoder-decoder network to map nucleus representations (like distance maps and centroid probability maps) to other shape representations (e.g., bounding boxes). The encoder of this trained network then acts as a feature extractor, and the SAP loss penalizes differences in these extracted shape features between predicted and ground-truth nuclei.
  • Fine-grained Post-Processing (FPP): An improved post-processing pipeline is proposed to refine the segmentation masks by incorporating semantic segmentation information to correct false positives and false negatives.

Datasets

The paper evaluates CPP-Net on three publicly available datasets:

  • DSB2018: The Data Science Bowl 2018 dataset, a benchmark for nucleus detection and segmentation.
  • BBBC006v1: A dataset containing U2OS cells.
  • PanNuke: A dataset of H&E stained tissue images, featuring nuclei classified into different types.

The authors also perform cross-dataset evaluations to assess generalization ability.

Key Insights and Ideas

  • Context is King (Even for Nuclei): The core idea is that relying on a single centroid pixel is insufficient for accurate boundary prediction. By sampling multiple points around the centroid, the model gains a richer understanding of the local context, significantly improving distance estimations.
  • Not All Context is Equal: Simply averaging predictions from sampled points isn't optimal. CWM's adaptive weighting based on prediction confidence is crucial for effectively leveraging this context, discarding less reliable information.
  • Shape Matters: Explicitly enforcing nucleus shape constraints through the SAP loss is a powerful way to improve segmentation quality, especially for distinguishing complex or overlapping instances. The idea of using a pre-trained feature extractor to capture shape information is a clever application of perceptual loss principles.
  • Post-processing Refinement: The FPP module demonstrates that even with a strong deep learning model, careful post-processing can further boost performance by leveraging complementary information like semantic segmentation.

Key Results

CPP-Net achieves state-of-the-art performance on the DSB2018, BBBC006v1, and PanNuke datasets.

  • Accuracy Improvements: On DSB2018, CPP-Net improves the mean Average Precision (AP) by 3.45% and the Panoptic Quality (PQ) by 2.07% compared to the baseline StarDist model. Similar significant improvements are observed on BBBC006v1.
  • Robustness: The model shows strong performance across various AP thresholds, particularly at higher IoU thresholds (e.g., AP0.9), indicating better boundary localization.
  • Efficiency: While introducing new modules, CPP-Net remains computationally efficient, with inference times comparable to or only slightly higher than StarDist, and significantly faster than many other methods.
  • Generalization: Cross-dataset evaluations show that CPP-Net generalizes well to unseen datasets.

Limitations and Future Work

  • Polygon Assumption: CPP-Net, like StarDist, assumes nuclei can be represented by convex polygons. This might limit its performance on nuclei with highly irregular shapes. The authors suggest exploring spline representations as a future direction.
  • Broader Applications: The authors believe CPP-Net's ability to aggregate context and improve shape quality makes it suitable for other medical image analysis tasks, such as nucleus classification and segmentation of other complex structures like brain tumors and pancreases.

Summary updated

Abstract

Nucleus segmentation is a challenging task due to the crowded distribution and blurry boundaries of nuclei. Recent approaches represent nuclei by means of polygons to differentiate between touching and overlapping nuclei and have accordingly achieved promising performance. Each polygon is represented by a set of centroid-to-boundary distances, which are in turn predicted by features of the centroid pixel for a single nucleus. However, using the centroid pixel alone does not provide sufficient contextual information for robust prediction and thus degrades the segmentation accuracy. To handle this problem, we propose a Context-aware Polygon Proposal Network (CPP-Net) for nucleus segmentation. First, we sample a point set rather than one single pixel within each cell for distance prediction. This strategy substantially enhances contextual information and thereby improves the robustness of the prediction. Second, we propose a Confidence-based Weighting Module, which adaptively fuses the predictions from the sampled point set. Third, we introduce a novel Shape-Aware Perceptual (SAP) loss that constrains the shape of the predicted polygons. Here, the SAP loss is based on an additional network that is pre-trained by means of mapping the centroid probability map and the pixel-to-boundary distance maps to a different nucleus representation. Extensive experiments justify the effectiveness of each component in the proposed CPP-Net. Finally, CPP-Net is found to achieve state-of-the-art performance on three publicly available databases, namely DSB2018, BBBC06, and PanNuke. Code of this paper is available at \url{https://github.com/csccsccsccsc/cpp-net
BibTeX
@article{chen2021cpp,
  author = {Shengcong Chen and Changxing Ding and Minfeng Liu and Dacheng Tao},
  title = {{CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  doi = {10.1109/tip.2023.3237013},
}
Figure