TopoSeg: Topology-Aware Nuclear Instance Segmentation

Hongliang He, Jun Wang, Pengxu Wei, Fan Xu, Xiangyang Ji, Chang Liu, Jie ChenICCV 2023 · DOI

Abstract

Accurate nuclear instance segmentation is critical for digital pathology, yet standard deep learning models often struggle with dense or overlapping nuclei due to a lack of structural awareness. We present TopoSeg, a novel framework that incorporates topological constraints into the training process. By leveraging persistent homology and an adaptive selection strategy, TopoSeg ensures biologically plausible segmentations with maintained connectivity and boundary closure. Our approach achieves state-of-the-art performance on multiple benchmarks, demonstrating superior robustness in separating touching nuclei compared to traditional pixel-wise methods.

Introduction

In digital pathology, the precise segmentation of individual cell nuclei serves as the foundation for cancer diagnosis and prognosis. However, nuclei are frequently tightly packed or overlapping, presenting a significant challenge for automated systems. Most contemporary deep learning models rely on pixel-wise loss functions, such as Cross-Entropy or Dice loss, which treat pixels independently. Consequently, these models lack an inherent understanding of the global structure, often leading to "broken" boundaries or merged instances in crowded regions. Such topological errors compromise morphological measurements and clinical assessments.

TopoSeg addresses this gap by forcing the network to respect the underlying topology of the tissue. By integrating topological features directly into the loss function, the model learns to prioritize structural integrity—ensuring that each nucleus is represented as a single, connected object with a continuous boundary.

Methodology

The TopoSeg framework introduces two primary components to the standard segmentation pipeline: the Topology-Aware Module (TAM) and the Adaptive Topology-Aware Selection (ATS) strategy.

Topology-Aware Module (TAM)

The TAM quantifies the structural correctness of predicted segmentations using persistent homology. This mathematical tool encodes the dynamic evolution of topological features—such as connected components and holes—into persistence barcodes as the probability map is thresholded. By comparing the barcodes of the predicted maps against the ground truth, the TAM generates a differentiable topology-aware loss. This loss penalizes the model for high-level structural errors, such as splitting a single nucleus or failing to close a boundary, which standard pixel-wise losses might ignore.

Adaptive Topology-Aware Selection (ATS)

Calculating topological features is computationally intensive. To maintain training efficiency, the ATS strategy identifies "topology-important" regions. It adaptively selects specific images and hierarchical patches exhibiting high topological errors (measured via Betti numbers) and prioritizes them for optimization. This ensures the model focuses its computational resources on the most challenging, crowded areas of the slide where topological errors are most likely to occur.

The overall architecture utilizes a dual-branch U-Net: one branch handles semantic categories, while a "three-class" branch (predicting inside, boundary, and background) feeds into the TAM for topological refinement.

Experimental Results

TopoSeg was evaluated on three diverse nuclear instance segmentation datasets: MoNuSeg (multi-organ), CPM17 (MICCAI challenge), and PBNuclei (clinical data from Peking University Affiliated Hospital).

Quantitative Performance

The model achieved state-of-the-art results across all benchmarks, particularly in metrics sensitive to instance-level accuracy:

  • MoNuSeg: Achieved a Panoptic Quality (PQ) of 0.625 and an Aggregated Jaccard Index (AJI) of 0.643, outperforming baselines like Hover-Net.
  • CPM17: Reached a PQ of 0.705 and AJI of 0.756.

Qualitative Impact of Topological Loss

Visual inspections confirm that TopoSeg is significantly more robust at separating touching nuclei. While traditional models often produce merged blobs in dense clusters, TopoSeg maintains distinct, closed boundaries.

Comparison of pixel-wise loss vs. topo-aware loss. The topo-aware loss successfully separates merged nuclei by focusing on the 'probability trend' and structural integrity, reducing topological errors to zero even if pixel-wise error slightly increases.

A key insight from our work is that a few incorrect pixels at a boundary can have a catastrophic impact on topology (e.g., merging two objects), whereas many incorrect pixels inside a large nucleus might not change the structure at all. TopoSeg effectively prioritizes fixing these critical boundary pixels.

Conclusion and Future Work

TopoSeg demonstrates that incorporating topology as a differentiable constraint significantly improves the biological plausibility of nuclear segmentations. By focusing on "hard" patches through the ATS module, the framework achieves high performance without prohibitive computational costs. Future work will explore scaling this approach to massive whole-slide images (WSIs) and extending the topology-aware framework to 3D medical imaging modalities such as CT or MRI, where connectivity is equally paramount.

Summary updated

BibTeX
@inproceedings{he2023toposeg,
  author = {Hongliang He and Jun Wang and Pengxu Wei and Fan Xu and Xiangyang Ji and Chang Liu and Jie Chen},
  title = {{TopoSeg: Topology-Aware Nuclear Instance Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01948},
}
Figure