Deep Learning-based Automatic Assessment of AgNOR-scores in Histopathology Images

Jonathan Ganz, Karoline Lipnik, Jonas Ammeling, Barbara Richter, Chloé Puget, Eda Parlak, Laura Diehl, Robert Klopfleisch, Taryn A. Donovan, Matti Kiupel, Christof A. Bertram, Katharina Breininger, Marc Aubreville

TL;DR

This paper introduces a deep learning pipeline to automatically quantify AgNOR scores in histopathology images, a crucial but time-consuming task for cancer prognosis. The developed system achieves performance comparable to human pathologists, significantly reducing the manual effort required.

What Problem Does This Paper Address?

Histopathological analysis often relies on quantifying specific cellular features to predict disease outcomes. Nucleolar Organizer Regions (NORs), visualized as Argyrophilic NORs (AgNORs) after silver staining, are one such feature. The number of AgNORs per nucleus is a known prognostic factor for various tumors, correlating with cell proliferation and tumor growth. However, manually counting AgNORs in hundreds of nuclei per slide is a laborious and time-intensive process for pathologists. This paper aims to automate this process using deep learning, addressing the need for faster, more consistent, and scalable AgNOR scoring in clinical settings.

Methods

The authors frame the AgNOR scoring problem as an object detection task, focusing on the quantity of AgNORs within a nucleus rather than precise segmentation of each individual AgNOR. This approach simplifies the annotation process. They employ an anchor-free, fully convolutional one-stage object detector (FCOS) for this task. To handle the class imbalance in their dataset (where nuclei with few AgNORs are far more common than those with many), they use an ensemble of five detection models trained on different data subsets. For training, they oversample minority classes and use small image patches (128x128 pixels) to better control the class distribution within each patch. The models are trained using the Adam optimizer with a learning rate of 10⁻⁴. For evaluation, the inference process mirrors the manual annotation procedure, dividing images into fields and processing them sequentially.

Datasets

The study utilizes 29 images of canine cutaneous mast cell tumors (CCMCT) stained for AgNORs. These images, each 1569x1177 pixels, were selected by a pathologist from whole slide images and digitized at a resolution of 0.25 microns per pixel. All nuclei within these images were annotated by a pathologist, with the number of identifiable AgNORs serving as the label for each nucleus. This resulted in 23,036 bounding box annotations across twelve classes (including a class for >10 AgNORs). Ten of these images were set aside as a hold-out test set.

Key Insights and Ideas

  • Object Detection over Segmentation: Framing AgNOR scoring as an object detection problem (counting) rather than a segmentation problem (outlining each AgNOR) significantly reduces the annotation burden while still capturing the essential prognostic information.
  • Ensemble of Detectors for Robustness: Using an ensemble of five FCOS models trained on different data subsets helps to improve the overall robustness and performance of the AgNOR scoring pipeline, especially in the face of data imbalances.
  • Class Imbalance Mitigation: The strategies of oversampling minority classes during training and using smaller patch sizes are crucial for effectively training a model on a dataset where the distribution of AgNOR counts is heavily skewed.
  • Human-like Performance: The study demonstrates that deep learning models can achieve performance comparable to human experts in this specific histopathological task, opening doors for widespread automation.

Key Results

The developed deep learning pipeline achieved a mean squared error (MSE) of 0.054 and a mean absolute error (MAE) of 0.201 when comparing its AgNOR scores to those of six human pathologists across all images and raters. This indicates a high degree of agreement with expert assessments. At the object level, the model showed satisfactory performance in detecting nuclei with up to 2 AgNORs, achieving mean F1-scores greater than 60%. However, detection accuracy decreased for classes with more than 2 AgNORs per nucleus, with mean F1-scores below 40%, likely due to the scarcity of such instances in the training data.

Limitations and Future Work

The authors acknowledge that the manual annotation process, which involved focusing through individual nuclei at a single z-plane, might not perfectly replicate the full 3D nature of the tissue. They suggest that using image stacks at multiple focus levels and higher scan resolutions could potentially improve the ability of both humans and algorithms to distinguish individual AgNORs. The reduced performance on higher AgNOR count classes is attributed to the significant class imbalance in the training data. Future work could focus on further improving detection for these rarer classes. The authors encourage the use of such automated pipelines for future clinical research to investigate the prognostic significance of AgNORs.

Summary updated

Abstract

Nucleolar organizer regions (NORs) are parts of the DNA that are involved in RNA transcription. Due to the silver affinity of associated proteins, argyrophilic NORs (AgNORs) can be visualized using silver-based staining. The average number of AgNORs per nucleus has been shown to be a prognostic factor for predicting the outcome of many tumors. Since manual detection of AgNORs is laborious, automation is of high interest. We present a deep learning-based pipeline for automatically determining the AgNOR-score from histopathological sections. An additional annotation experimentwas conducted with six pathologists to provide an independent performance evaluation of our approach. Across all raters and images, we found a mean squared error of 0.054 between the AgNORscores of the experts and those of the model, indicating that our approach offers performance comparable to humans.
BibTeX
@inproceedings{ganz2023deep,
  author = {Jonathan Ganz and Karoline Lipnik and Jonas Ammeling and Barbara Richter and Chloé Puget and Eda Parlak and Laura Diehl and Robert Klopfleisch and Taryn A. Donovan and Matti Kiupel and Christof A. Bertram and Katharina Breininger and Marc Aubreville},
  title = {{Deep Learning-based Automatic Assessment of AgNOR-scores in Histopathology Images}},
  booktitle = {Bildverarbeitung für die Medizin},
  year = {2023},
  doi = {10.1007/978-3-658-41657-7\_49},
}
Figure