A Deep Learning Framework for Nuclear Segmentation and Classification in Histopathological Images

Sen Yang, Jinxi Xiang, Xiyue WangarXiv 2022 · DOI · arXiv

Abstract

Nucleus segmentation and classification are the prerequisites in the workflow of digital pathology processing. However, it is very challenging due to its high-level heterogeneity and wide variations. This work proposes a deep neural network to simultaneously achieve nuclear classification and segmentation, which is designed using a unified framework with three different branches, including segmentation, HoVer mapping, and classification. The segmentation branch aims to generate the boundaries of each nucleus. The HoVer branch calculates the horizontal and vertical distances of nuclear pixels to their centres of mass. The nuclear classification branch is used to distinguish the class of pixels inside the nucleus obtained from segmentation.

Introduction

In digital pathology, accurately identifying the boundaries (segmentation) and types (classification) of cell nuclei is a fundamental step for diagnosis and research. However, this task is notoriously difficult due to the high heterogeneity of nuclei across different tissues, the frequent overcrowding of cells that leads to overlapping instances, and significant class imbalances where certain cell types appear much less frequently than others. This paper introduces an enhanced deep learning framework designed to address these challenges through a unified multi-branch architecture.

Methodology

The proposed framework is built on a modified U-Net architecture, incorporating several sophisticated components to enhance spatial awareness and feature extraction.

Three-Branch Architecture

As illustrated in the architecture diagram, the network splits into three distinct "heads":

  1. Segmentation Branch: Predicts the general boundaries of nuclei.
  2. HoVer Branch: Generates horizontal and vertical distance maps, calculating the distance of each pixel from the nucleus's center of mass. This is critical for separating touching nuclei during post-processing.
  3. Classification Branch: Categorizes pixels into six types, including epithelial, lymphocyte, plasma, eosinophil, neutrophil, and connective tissue.
The proposed multi-branch U-Net architecture incorporating SE-Res and Coordinate Attention (CA) modules for simultaneous segmentation, HoVer mapping, and classification.

Advanced Modules

To improve performance, the authors replaced standard convolutions in the encoder with Squeeze-and-Excitation Residual (SE-Res) modules, which use a gating mechanism to emphasize informative features. In the decoder, Coordinate Attention (CA) modules are embedded to capture both cross-channel relationships and direction-specific spatial information, improving localization accuracy. To combat class imbalance, the classification branch utilizes a Weighted Cross-Entropy (LWCE) loss, assigning higher importance to rarer cell types.

Experimental Results

The model was evaluated using the CoNIC (Colon Nuclei Identification and Counting) Challenge 2022 dataset. The results demonstrate that the integration of attention mechanisms and multi-branch learning allows the model to capture subtle textures necessary for distinguishing similar cell types.

Quantitative comparison of the proposed method against HoVer-Net and MaskRCNN on the CoNIC dataset.

As shown in the performance table, the proposed model significantly outperforms established baselines:

  • mPQ+ (Multi-class Panoptic Quality): The model achieved a score of 0.434, a substantial improvement over HoVer-Net (0.295) and MaskRCNN (0.354).
  • $R^2$ (Coefficient of Determination): Measuring counting accuracy, the model reached 0.564, whereas the baseline HoVer-Net struggled with the dataset's complexity, producing a negative value (-0.428).

Conclusion

By combining SE-Res and CA modules within a unified three-branch framework, this research provides a robust solution for nuclear instance segmentation and classification. While the model currently relies on specific post-processing for distance maps and has been primarily validated on colon tissue, its superior performance on the CoNIC dataset marks a significant advancement in automated histopathological analysis. Future work may focus on streamlining the post-processing into an end-to-end trainable system and testing generalizability across other tissue types.

Summary updated

Abstract

Nucleus segmentation and classification are the prerequisites in the workflow of digital pathology processing. However, it is very challenging due to its high-level heterogeneity and wide variations. This work proposes a deep neural network to simultaneously achieve nuclear classification and segmentation, which is designed using a unified framework with three different branches, including segmentation, HoVer mapping, and classification. The segmentation branch aims to generate the boundaries of each nucleus. The HoVer branch calculates the horizontal and vertical distances of nuclear pixels to their centres of mass. The nuclear classification branch is used to distinguish the class of pixels inside the nucleus obtained from segmentation.
BibTeX
@misc{yang2022deep,
  author = {Sen Yang and Jinxi Xiang and Xiyue Wang},
  title = {{A Deep Learning Framework for Nuclear Segmentation and Classification in Histopathological Images}},
  year = {2022},
  eprint = {2203.03420},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.03420},
}
Figure