Nuclear pleomorphism in canine cutaneous mast cell tumors: Comparison of reproducibility and prognostic relevance between estimates, manual morphometry, and algorithmic morphometry

Andreas Haghofer, Cem Parlak, Alexander Bartel, Taryn Donovan, Charles‐Antoine Assenmacher, Pompei Bolfă, Michael J. Dark, Andrea Fuchs‐Baumgartinger, Andrea Klang, Kathrin Jäger, Robert Klopfleisch, Sophie Merz, Barbara Richter, F. Y. Schulman, Hannah Janout, Jonathan Ganz, Josef Scharinger, Marc Aubreville, Stephan Winkler, Matti Kiupel, Christof Bertram

TL;DR

This study tackles the challenge of inconsistent pathological assessments of nuclear pleomorphism in canine mast cell tumors by comparing traditional subjective estimates with quantitative morphometric approaches. They found that both a simplified manual morphometry method and an automated deep learning approach significantly improve reproducibility and prognostic accuracy compared to subjective pathologist estimates, paving the way for more reliable cancer diagnostics.

What Problem Does This Paper Address?

Diagnosing and predicting the behavior of tumors often relies on microscopic examination of cells, particularly the variation in nuclear size and shape (anisokaryosis and pleomorphism). However, subjective assessments by pathologists, while common, suffer from poor reproducibility between different individuals and even by the same individual at different times. This inconsistency can lead to variations in diagnosis and treatment recommendations. While manual morphometry (measuring nuclei) offers a more objective approach, traditional methods requiring the measurement of at least 100 nuclei are too time-consuming for routine clinical practice. This paper aims to bridge this gap by evaluating more practical and automated methods for quantifying nuclear characteristics in canine cutaneous mast cell tumors (ccMCTs), a common and potentially aggressive cancer in dogs.

Methods

The researchers compared four methods for evaluating nuclear characteristics:

  1. Anisokaryosis Estimates: 11 pathologists provided subjective assessments of nuclear variation using two different classification schemes (2-tier and 3-tier).
  2. Gold Standard Manual Morphometry: One pathologist manually measured at least 100 nuclei per tumor to establish a benchmark for accuracy and prognostic value.
  3. Practicable Manual Morphometry: Nine pathologists measured a smaller, stratified sample of 12 nuclei per tumor. This method aimed for a balance between accuracy and practicality.
  4. Algorithmic Morphometry: A deep learning-based segmentation model was developed to automatically identify and measure all nuclei within digital images of the tumors.

The study evaluated these methods based on their accuracy, reproducibility (inter-rater and intra-rater), and prognostic relevance (predicting tumor-specific survival).

Datasets

The study utilized two distinct datasets of canine cutaneous mast cell tumors (ccMCTs):

  1. Outcome Data Set: This dataset comprised 96 ccMCT cases with associated patient survival outcome information. This was primarily used for evaluating the prognostic value and reproducibility of the different methods.
  2. Ground Truth Data Set: This dataset contained annotated images of ccMCT nuclei, where the precise outlines of individual nuclei were meticulously marked by an expert. This dataset was crucial for training, validating, and testing the deep learning-based algorithmic morphometry model, as well as for assessing measurement accuracy.

The paper does not explicitly mention introducing a new benchmark dataset, but rather uses existing ccMCT samples to compare established and novel evaluation methods.

Key Insights and Ideas

  • Subjectivity is a Major Hurdle: The study strongly confirms that subjective estimates of nuclear pleomorphism are highly variable between pathologists, making them unreliable for consistent clinical decision-making.
  • Practicable Morphometry is Viable: A simplified manual morphometry approach, measuring just 12 nuclei with stratified sampling, offers a significant improvement in reproducibility and prognostic power over subjective estimates, making it a feasible option for routine diagnostics.
  • AI for Precision Pathology: Deep learning-based algorithmic morphometry demonstrated the highest accuracy and prognostic value, highlighting the potential of AI to revolutionize quantitative pathology by analyzing thousands of cells efficiently and consistently.
  • Nuclear Size Matters Most: Parameters related to nuclear size (like standard deviation of nuclear area) consistently showed better prognostic value than those related to nuclear shape across all tested methods.
  • Tumor Heterogeneity is a Factor: Preliminary findings suggest that intratumoral variation in nuclear features can also have prognostic implications, opening avenues for further research into optimal sampling strategies.

Key Results

  • Reproducibility:
    • Inter-rater reproducibility for anisokaryosis estimates was low (kappa = 0.226).
  • Practicable manual morphometry of the standard deviation (SD) of nuclear size showed good inter-rater reproducibility (ICC = 0.654).
  • Prognostic Value (Tumor-Specific Survival):
    • Gold standard manual morphometry (≥100 nuclei) achieved an AUC of 0.839.
  • Practicable manual morphometry (12 nuclei) achieved a high AUC of 0.868.
  • Algorithmic morphometry achieved the highest AUC of 0.943.
  • The independent benchmark, Mitotic Count (MC), had AUCs ranging from 0.885 to 0.900, with morphometric size parameters performing comparably or better.
  • Accuracy: Algorithmic morphometry showed a smaller measurement error compared to the ground truth annotations than practicable manual morphometry.

Limitations and Future Work

The authors acknowledge several limitations:

  • Lack of Cause of Death Confirmation: For some cases in the outcome data set, the exact cause of death was not definitively confirmed, potentially introducing bias.
  • Limited Number of Tumor-Related Deaths: The study had a relatively small number of cases with tumor-related deaths, which hindered multivariate analysis and the exploration of combinations of prognostic factors.
  • Need for Validation: The findings, particularly regarding prognostic value and tumor heterogeneity, require validation in larger, independent study populations.
  • Inter-Algorithmic Reproducibility: While their algorithm showed high reproducibility, the consistency between different algorithmic models needs to be investigated.
  • Algorithmic Robustness: The robustness of the algorithm to variations in image acquisition (e.g., different scanners) and tumor types needs further evaluation.
  • Tumor Heterogeneity Exploration: The study's preliminary findings on tumor heterogeneity are based on a limited number of regions. Future work should explore this more comprehensively across entire tumor sections.
  • Nuclear Shape Parameters: While nuclear size parameters showed strong prognostic value, shape parameters were less informative in this study and warrant further investigation, potentially in combination with size.

Future work should focus on validating these methods, exploring inter-algorithmic reproducibility, and delving deeper into the prognostic implications of tumor heterogeneity. The authors also suggest exploring pictorial illustrations or reference sizes to improve the reproducibility of subjective anisokaryosis estimates.

Summary updated

Abstract

Variation in nuclear size and shape is an important criterion of malignancy for many tumor types; however, categorical estimates by pathologists have poor reproducibility. Measurements of nuclear characteristics can improve reproducibility, but current manual methods are time-consuming. The aim of this study was to explore the limitations of estimates and develop alternative morphometric solutions for canine cutaneous mast cell tumors (ccMCTs). We assessed the following nuclear evaluation methods for accuracy, reproducibility, and prognostic utility: (1) anisokaryosis estimates by 11 pathologists; (2) gold standard manual morphometry of at least 100 nuclei; (3) practicable manual morphometry with stratified sampling of 12 nuclei by 9 pathologists; and (4) automated morphometry using deep learning-based segmentation. The study included 96 ccMCTs with available outcome information. Inter-rater reproducibility of anisokaryosis estimates was low (k = 0.226), whereas it was good (intraclass correlation = 0.654) for practicable morphometry of the standard deviation (SD) of nuclear size. As compared with gold standard manual morphometry (area under the ROC curve [AUC] = 0.839, 95% confidence interval [CI] = 0.701-0.977), the prognostic value (tumor-specific survival) of SDs of nuclear area for practicable manual morphometry and automated morphometry were high with an AUC of 0.868 (95% CI = 0.737-0.991) and 0.943 (95% CI = 0.889-0.996), respectively. This study supports the use of manual morphometry with stratified sampling of 12 nuclei and algorithmic morphometry to overcome the poor reproducibility of estimates. Further studies are needed to validate our findings, determine inter-algorithmic reproducibility and algorithmic robustness, and explore tumor heterogeneity of nuclear features in entire tumor sections.
BibTeX
@inproceedings{haghofer2024nuclear,
  author = {Andreas Haghofer and Cem Parlak and Alexander Bartel and Taryn Donovan and Charles‐Antoine Assenmacher and Pompei Bolfă and Michael J. Dark and Andrea Fuchs‐Baumgartinger and Andrea Klang and Kathrin Jäger and Robert Klopfleisch and Sophie Merz and Barbara Richter and F. Y. Schulman and Hannah Janout and Jonathan Ganz and Josef Scharinger and Marc Aubreville and Stephan Winkler and Matti Kiupel and Christof Bertram},
  title = {{Nuclear pleomorphism in canine cutaneous mast cell tumors: Comparison of reproducibility and prognostic relevance between estimates, manual morphometry, and algorithmic morphometry}},
  booktitle = {Veterinary},
  year = {2024},
  doi = {10.1177/03009858241295399},
}
Figure