Cytologic Scoring of Equine Exercise-Induced Pulmonary Hemorrhage (EIPH): Performance of Human Experts and a Deep Learning-Based Algorithm

Christof Bertram, Christian Marzahl, Alexander Bartel, Jason Stayt, Federico Bonsembiante, Janet Beeler‐Marfisi, Ann Kristin Barton, Ginevra Brocca, Maria Elena Gelain, Agnes C. Gläsel, Kelly du Preez, Kristina Weiler, Christiane Weissenbacher‐Lang, Katharina Breininger, Marc Aubreville, Andreas Maier, Robert Klopfleisch, Jenny HillbioRxiv 2022 · DOI

TL;DR

This paper tackles the challenge of accurately diagnosing Exercise-Induced Pulmonary Hemorrhage (EIPH) in horses, a condition diagnosed by scoring hemosiderin in lung cells. They found human experts struggle with consistency, leading to a proposed uncertainty interval for diagnosis. A deep learning algorithm, however, achieved higher accuracy and reproducibility, suggesting its potential to revolutionize this diagnostic process.

What Problem Does This Paper Address?

Diagnosing Exercise-Induced Pulmonary Hemorrhage (EIPH) in sport horses is crucial for their performance and well-being. A key diagnostic method involves examining bronchoalveolar lavage fluid (BALF) cells for hemosiderin, a byproduct of red blood cell breakdown. This is quantified using a Total Hemosiderin Score (THS). However, the manual scoring of these cells by human experts is time-consuming and suffers from significant inter-observer variability, meaning different experts can arrive at different scores for the same sample. This inconsistency makes routine diagnosis and large-scale studies challenging. The authors aim to address this by evaluating the performance of human experts and a novel deep learning-based algorithm for THS scoring.

Methods

The study employed a multi-pronged approach:

  • Human Annotation: Ten trained annotators, including board-certified pathologists and trainees, scored digitized, iron-stained BALF slides using a standardized method. They used specialized software for viewing whole slide images (WSIs) and annotating individual macrophages.
  • Deep Learning Algorithm: A state-of-the-art object detection network, RetinaNet, was trained on a ground truth dataset. This algorithm was designed to automatically identify and score hemosiderin content in macrophages across entire WSIs.
  • Ground Truth Dataset: A meticulously created ground truth dataset was developed through a multi-step process involving expert annotations, algorithmic predictions, and rigorous review to minimize human error and bias.
  • Reference Methods: To validate the THS scores, the researchers also measured RBC count, hemoglobin concentration, and iron concentration in BALF samples.

Datasets

  • Study Specimens: 52 equine BALF samples were collected and processed into 104 cytological specimens (two per sample).
  • Ground Truth Dataset: This dataset, containing annotations for all alveolar macrophages in the 52 WSIs, was developed using a multi-step, computer-assisted labeling approach. It was also used to train the deep learning models.

Key Insights and Ideas

  • Human Variability is a Major Hurdle: The study strongly highlights that even trained experts exhibit significant variability in scoring hemosiderin. This isn't just random error; a systematic bias in how they interpret and grade the iron pigment is a primary driver.
  • Standardization is Key, But Difficult: While standardizing grading (e.g., using a color chart) could help human experts, the inherent subjectivity in interpreting color saturation makes it challenging.
  • Deep Learning Offers a Solution: The deep learning algorithm demonstrated remarkable consistency and accuracy, closely mirroring the ground truth and outperforming individual human annotators in diagnostic accuracy.
  • Iron Concentration as a Potential Biomarker: The study found a significant correlation between iron concentration in BALF and THS, suggesting it could be a valuable, objective measure for assessing pulmonary bleeding, especially for chronic cases.

Key Results

  • Inter-Observer Variability: The interquartile range of the difference between annotators' THS was 30 score points. The intraclass correlation coefficient (ICC) for the THS was 0.685, indicating that 68.5% of the scoring variance was due to systematic error between annotators.
  • Diagnostic Accuracy:
    • Human annotators achieved an average diagnostic accuracy of 75.7% compared to the ground truth.
  • The deep learning algorithm achieved a significantly higher accuracy of 92.3% compared to the ground truth.
  • Speed: The deep learning algorithm could analyze thousands of cells in under two minutes, a stark contrast to the time-consuming manual annotation process.
  • Uncertainty Interval: For human expert diagnosis, an 80% uncertainty interval of 40-110 THS points is proposed, within which diagnosis is considered unreliable and not reproducible.
  • Correlation with Iron: Iron concentration showed a high correlation (R=0.75-0.79) with THS scores from both annotators and the algorithm, while RBC and hemoglobin concentrations did not.

Limitations and Future Work

  • Lack of a True Gold Standard: The authors acknowledge that a definitive "gold standard" for EIPH diagnosis is still lacking, which limits the absolute validation of both human and algorithmic performance.
  • Ground Truth Bias: Despite efforts to mitigate it, human error in creating the ground truth dataset could still introduce bias in performance evaluation.
  • WSI Limitations: The use of Whole Slide Images (WSIs) presented challenges, including limited resolution and variations in color representation between scanners, which could affect both human and algorithmic scoring.
  • Future Work:
    • Further studies are needed to determine the value of iron concentration as a reference method for THS and as a diagnostic test for EIPH.
  • Investigating how THS algorithms are best implemented in diagnostic workflows to ensure reliability is crucial.
  • Developing standardized "color charts" with clear thresholds for hemosiderin grading could improve human expert consistency.
  • Future research should critically evaluate potential sources of algorithmic errors before routine diagnostic use.

Summary updated

Abstract

Abstract Exercise-induced pulmonary hemorrhage (EIPH) is a relevant respiratory disease in sport horses which can be diagnosed by examination of bronchoalveolar lavage fluid (BALF) cells using the total hemosiderin score (THS). The aim of this study was to evaluate the diagnostic accuracy and reproducibility of trained annotators and to validate a deep learning-based algorithm for the THS. Digitized, iron-stained cytological specimens were prepared from 52 equine BALF samples. Ten annotators produced a THS for each slide according to published methods. The reference methods for comparing annotator’s and algorithmic performance included a ground truth dataset, the mean annotators’ THSs, and chemical iron measurements. Results of the study showed that annotators had marked inter-observer variability of the THS, which was mostly due to a systematic error between annotators in grading the intracytoplasmatic hemosiderin content of individual macrophages. Regarding overall measurement error between the annotators, 87.7% of the variance of the could be reduced by using standardized grades based on the ground truth. The algorithm was highly consistent with the ground truth in assigning hemosiderin grades. Compared to the ground truth THS, annotators had an accuracy of diagnosing EIPH (THS of < or ≥ 75) of 75.7% whereas the algorithm had an accuracy of 92.3% with no marked difference in correlation to chemical iron measurements. The results show that deep learning-based algorithms are useful for improving reproducibility and routine applicability of the THS. For THS by experts, a diagnostic uncertainty interval of 40 to 110 is proposed. THSs within this interval have insufficient reproducibility regarding the EIPH diagnosis.
BibTeX
@misc{bertram2022cytologic,
  author = {Christof Bertram and Christian Marzahl and Alexander Bartel and Jason Stayt and Federico Bonsembiante and Janet Beeler‐Marfisi and Ann Kristin Barton and Ginevra Brocca and Maria Elena Gelain and Agnes C. Gläsel and Kelly du Preez and Kristina Weiler and Christiane Weissenbacher‐Lang and Katharina Breininger and Marc Aubreville and Andreas Maier and Robert Klopfleisch and Jenny Hill},
  title = {{Cytologic Scoring of Equine Exercise-Induced Pulmonary Hemorrhage (EIPH): Performance of Human Experts and a Deep Learning-Based Algorithm}},
  year = {2022},
  doi = {10.1101/2022.02.28.482322},
  howpublished = {bioRxiv},
}
Figure