Mutual-Complementing Framework for Nuclei Detection and Segmentation in Pathology Image

Zunlei Feng, Zhonghua Wang, Xinchao Wang, Yining Mao, Thomas Li, Jie Lei, Yuexuan Wang, Mingli Song

Introduction

Digital pathology relies heavily on the accurate identification and segmentation of cell nuclei for cancer diagnosis and grading. While deep learning has significantly advanced this field, existing supervised methods require extensive manual annotations, which are costly and time-consuming. Conversely, unsupervised methods often struggle with "degenerated" cancerous tissues where nuclei overlap or exhibit irregular morphologies. This paper introduces the Mutual-Complementing Framework (MCF), designed to achieve high-accuracy nuclei detection and segmentation using only a minimal set of initial templates.

The Mutual-Complementing Framework (MCF)

The MCF operates through a synergistic feedback loop between a detection branch and a segmentation branch, allowing the system to iteratively refine its performance without massive labeled datasets.

Detection and Segmentation Branches

The Detection Branch utilizes correlation filters (CF) to identify patterns matching a few initial nucleus templates. To handle varying scales and dense clusters, it employs Multi-scale and Cropped Filter Fusion (MFF & CFF). A key component is Gradient Direction Based Postprocessing (GDBP), which accurately separates overlapping nuclei by analyzing outward-pointing gradients from nucleus centers.

The Segmentation Branch consists of a U-Net architecture. Initially, it uses the high-confidence response maps from the detection branch as pseudo-masks for training.

The Mutual-Complementing Framework (MCF) architecture, illustrating the iterative feedback loop between detection and segmentation branches.

The Feedback Loop

The core innovation lies in the bidirectional information flow:

  1. Detection → Segmentation: The detection branch provides initial targets for the CNN to learn.
  2. Segmentation → Detection: The CNN identifies nuclei missed by the initial filters. By calculating the "Mask Residual"—the difference between CNN predictions and filter detections—the system extracts new templates to update the detection branch for subsequent iterations.

Experimental Evaluation

The framework was validated on several public datasets (TNBC, MICCAI18, NuCLS) and a custom benchmark of 50 degenerated cancerous samples.

Key Results and Robustness

MCF demonstrates remarkable efficiency, achieving performance comparable to fully supervised models like Mask R-CNN and U-Net using only 10–16 initial templates. Crucially, MCF maintains high performance on difficult, degenerated samples where traditional unsupervised methods typically fail. While other methods saw performance drops of over 20% on messy clinical samples, MCF's performance remained stable with only a ~6% decrease.

Qualitative comparison of nuclei segmentation results across various methods. MCF shows superior performance, particularly in maintaining accuracy on complex cancerous samples compared to other unsupervised and supervised baselines.

Conclusion

The Mutual-Complementing Framework bridges the gap between data-hungry supervised learning and less robust unsupervised methods. By leveraging the strengths of correlation filters and CNNs through a residual-based feedback loop, MCF provides a scalable and robust solution for nuclei analysis in complex pathological environments.

Summary updated

Abstract

Detection and segmentation of nuclei are fundamental analysis operations in pathology images, the assessments derived from which serve as the gold standard for cancer diagnosis. Manual segmenting nuclei is expensive and time-consuming. What’s more, accurate segmentation detection of nuclei can be challenging due to the large appearance variation, conjoined and overlapping nuclei, and serious degeneration of histological structures. Supervised methods highly rely on massive annotated samples. The existing two unsupervised methods are prone to failure on degenerated samples. This paper proposes a Mutual-Complementing Framework (MCF) for nuclei detection and segmentation in pathology images. Two branches of MCF are trained in the mutual-complementing manner, where the detection branch complements the pseudo mask of the segmentation branch, while the progressive trained segmentation branch complements the missing nucleus templates through calculating the mask residual between the predicted mask and detected result. In the detection branch, two response map fusion strategies and gradient direction based postprocessing are devised to obtain the optimal detection response. Furthermore, the confidence loss combined with the synthetic samples and self-finetuning is adopted to train the segmentation network with only high confidence areas. Extensive experiments demonstrate that MCF achieves comparable performance with only a few nucleus patches as supervision. Especially, MCF possesses good robustness (only dropping by about 6%) on degenerated samples, which are critical and common cases in clinical diagnosis.
BibTeX
@inproceedings{feng2021mutual,
  author = {Zunlei Feng and Zhonghua Wang and Xinchao Wang and Yining Mao and Thomas Li and Jie Lei and Yuexuan Wang and Mingli Song},
  title = {{Mutual-Complementing Framework for Nuclei Detection and Segmentation in Pathology Image}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00400},
}
Figure