Explainable Severity ranking via pairwise n-hidden comparison: a case study of glaucoma

Hong Nguyen, Cuong V. Nguyen, Shrikanth Narayanan, Benjamin Y. Xu, Michael PazzaniarXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces a novel framework for ranking glaucoma severity using fundus images by comparing pairs of images through "n-hidden comparisons" within a Siamese neural network. This approach not only improves diagnostic accuracy over traditional methods but also provides explainable AI (XAI) insights into why an image is deemed more severe, aiding clinical decision-making.

What Problem Does This Paper Address?

Accurately assessing the severity of primary open-angle glaucoma (POAG) is crucial for timely intervention and preventing vision loss. However, ophthalmologists can disagree on severity classifications due to varying individual thresholds. While they tend to agree when comparing two images side-by-side, existing AI models often focus on classifying images into discrete severity levels rather than learning these nuanced pairwise preferences. Furthermore, understanding why an AI model makes a particular severity judgment is often lacking, hindering clinical trust and adoption. This paper addresses the need for an AI system that can effectively rank glaucoma severity through pairwise comparisons and provide interpretable explanations for its decisions.

Methods

The core of the proposed method is a Siamese neural network adapted for severity ranking. Unlike standard Siamese networks that might output a single score for comparison, this work introduces "n-hidden comparisons." This means the network learns to compare images across n different "hidden" dimensions of feature space, rather than relying on a single aggregated severity score. This allows for a more nuanced understanding of severity, mirroring how ophthalmologists consider multiple factors (like optic cup-to-disc ratio, disc contour, and vessel position) when making judgments.

For the comparison itself, the paper leverages a modified RankNet architecture. The output of the n-hidden comparison is then fed into a fully connected network to predict the pairwise severity ranking. To explain these rankings, the authors employ SHAP (SHapley Additive exPlanations) to identify which of the n comparisons are most influential for a given decision. Subsequently, GradCAM is used to generate heatmaps highlighting the specific image regions that contribute to these influential comparisons, providing visual explanations.

Datasets

The study utilizes the Ocular Hypertension Treatment Study (OHTS) dataset. This dataset contains over 74,000 fundus images from multi-center prospective treatment trials for intraocular pressure (IOP) and POAG. For this research, the focus is on patients diagnosed with POAG. The dataset was curated to include patients who had multiple annual doctor visits, ensuring the availability of images that could represent different stages of the disease, including transitions from healthy to glaucomatous states. Two experimental settings were used:

  • Longitudinal setting: Compares two images from the same subject over time.
  • Cross-sectional setting: Compares image pairs without regard to subject identity.

Key Insights and Ideas

  • Beyond Single Scores: The "n-hidden comparison" approach is a significant departure from traditional single-score comparison methods. It acknowledges that severity is multi-faceted and can be better captured by learning multiple, distinct comparison dimensions.
  • Pairwise Preference is Key: The paper reinforces the idea that clinicians often find pairwise comparisons more reliable than absolute severity classifications, especially when there's ambiguity in defining exact "cut-off" points for severity levels.
  • Explainability Matters: Integrating SHAP and GradCAM for pairwise comparisons offers a more targeted and interpretable explanation than typical image-level XAI. It shows which specific comparative features led to the model's decision.
  • Longitudinal Data Advantage: The results suggest that longitudinal data (comparing images from the same patient over time) leads to significantly better performance than cross-sectional data, likely due to the inherent conceptual alignment.

Key Results

The proposed "n-hidden comparison" Siamese network demonstrated superior performance compared to baseline single-comparison models.

  • Accuracy Improvement: The 10-hidden comparison model achieved a 12% higher comparison accuracy than conventional single-comparison Siamese networks using ResNet and ViT backbones in the longitudinal setting.
  • Listwise Ranking: The model's performance was also competitive in listwise ranking (using nDCG), outperforming a baseline with the VGG19 backbone even without a dedicated list-rank reconstruction phase.
  • Qualitative Explanations: Visualizations (Figure 3) show that the model's saliency maps align with ophthalmologists' explanations, highlighting relevant areas like the expansion of the optic nerve and the optic disc rim. For instance, one explanation pointed to the optic nerve expanding towards the upper-right and lower-right, which was visually corroborated by the model's heatmaps.

Limitations and Future Work

  • Cross-sectional Uncertainty: The paper notes that cross-sectional comparisons are more challenging and less accurate than longitudinal ones. This is attributed to the lack of conceptual alignment between images from different subjects and the reliance solely on the MD-index.
  • Data Requirements: For cross-sectional comparisons to be more effective, the authors suggest the need for more extensive pairwise annotations that go beyond just the MD-index, incorporating other severity indicators.
  • Future Directions: The authors suggest using fewer comparisons when dealing with limited annotations and acquiring more annotations for cross-sectional comparison tasks. They also propose further exploration into refining the feature space dimension for improved severity ranking.

Summary updated

Abstract

Primary open-angle glaucoma (POAG) is a chronic and progressive optic nerve condition that results in an acquired loss of optic nerve fibers and potential blindness. The gradual onset of glaucoma results in patients progressively losing their vision without being consciously aware of the changes. To diagnose POAG and determine its severity, patients must undergo a comprehensive dilated eye examination. In this work, we build a framework to rank, compare, and interpret the severity of glaucoma using fundus images. We introduce a siamese-based severity ranking using pairwise n-hidden comparisons. We additionally have a novel approach to explaining why a specific image is deemed more severe than others. Our findings indicate that the proposed severity ranking model surpasses traditional ones in terms of diagnostic accuracy and delivers improved saliency explanations.
BibTeX
@misc{nguyen2023explainable,
  author = {Hong Nguyen and Cuong V. Nguyen and Shrikanth Narayanan and Benjamin Y. Xu and Michael Pazzani},
  title = {{Explainable Severity ranking via pairwise n-hidden comparison: a case study of glaucoma}},
  year = {2023},
  eprint = {2312.02541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.02541},
}
Figure