Distributional Shifts in Automated Diabetic Retinopathy Screening

Jay Nandy, Wynne Hsu, Mong-Li Lee

Abstract

Deep learning-based models are developed to automatically detect if a retina image is 'referable' in diabetic retinopathy (DR) screening. However, their classification accuracy degrades as the input images distributionally shift from their training distribution. Further, even if the input is not a retina image, a standard DR classifier produces a high confident prediction that the image is 'referable'. Our paper presents a Dirichlet Prior Network-based framework to address this issue. It utilizes an out-of-distribution (OOD) detector model and a DR classification model to improve generalizability by identifying OOD images. Experiments on real-world datasets indicate that the proposed framework can eliminate the unknown non-retina images and identify the distributionally shifted retina images for human intervention.

Introduction: The Challenge of Distributional Shift

Automated Diabetic Retinopathy (DR) screening using Deep Neural Networks (DNNs) has reached clinically acceptable accuracy in controlled environments. However, these models often "fail silently" when deployed in the real world due to distributional shift.

In a laboratory setting, a model performs well on data similar to its training set. In practice, models encounter images from different cameras, varying lighting conditions, or even non-retina images (e.g., blurry photos or natural scenes). Standard DNNs tend to be "over-confident" in these scenarios, often assigning a high-probability "referable" diagnosis to irrelevant images. This lack of "knowledge uncertainty" quantification makes it difficult for clinicians to trust fully automated systems.

Methodology: A Dirichlet-Based Framework

To address over-confidence, this research moves beyond standard Softmax classifiers by employing Dirichlet Prior Networks (DPN). Unlike standard models that predict a single probability distribution, a DPN predicts the parameters of a Dirichlet distribution, effectively representing a "distribution of distributions."

The proposed framework utilizes a DR classifier and an OOD detector to filter non-retinal images and flag shifted retinal images for human intervention.

As shown in the framework above, the authors propose a dual-model architecture:

  1. DR Classifier: Trained on in-domain retina images and natural images to improve general feature representation.
  2. OOD Detector: A separate model trained specifically to distinguish between "known" retina images, natural images, and a small set of "shifted" retina images from different sources.

Quantifying Uncertainty

The framework uses Mutual Information (MI) as a score for distributional uncertainty. A high MI score indicates the model is encountering something it hasn't seen before (Out-of-Distribution). This allows the system to categorize uncertainty into distinct types.

Dirichlet distributions on a simplex representing (a) confident prediction, (b) misclassification (data uncertainty), and (c) distributional shift (knowledge uncertainty).

Experimental Setup and Datasets

The study utilizes a diverse array of datasets to simulate real-world variety:

  • In-domain (Training): SiDRP (Singapore National DR Screening Program).
  • OOD Retina (Testing Shift): Kaggle DR dataset, Messidor, and Mayuri.
  • OOD Non-retina: ImageNet-Small, STL10, LSUN, and Texture datasets.
  • OOD Training Subset: A small slice of 1,200 images from the Kaggle dataset was used to help the OOD detector learn the characteristics of "shifted" retina images.

Key Results and Insights

  • Non-Retina Filtering: The OOD detector achieved 100% detection accuracy for non-retina datasets like STL10 and LSUN at a 5% threshold, whereas baseline models were essentially guessing (0.3% to 1.7% detection).
  • Handling Shifted Data: On the Messidor and Mayuri retina datasets, the proposed OOD detector outperformed the baseline by approximately 20% in identifying shifted images.
  • Improved Reliability: By discarding the top 10% of images with the highest uncertainty, the AUROC for the Kaggle-Test set improved from 83.7% to over 87%. This demonstrates that the system successfully identifies the samples it is most likely to misclassify.
  • Trust through Rejection: The core philosophy is that a model is more useful if it knows when to say "I don't know." Training a separate OOD detector proved more effective than relying on a single classifier to handle both diagnosis and shift detection.

Discussion and Limitations

While the framework significantly improves reliability, it currently requires a small set of "shifted" retina images during training to learn the nuances of distributional shift. Future work will focus on detecting these shifts in a completely unsupervised manner. Additionally, clinical deployment will require careful calibration of uncertainty thresholds ($\tau$) to balance the workload of human graders against the risk of automated misdiagnosis. Finally, the computational overhead of running two separate VGG-19 networks must be considered for high-throughput screening environments.

Summary updated

Abstract

Deep learning-based models are developed to automatically detect if a retina image is `referable' in diabetic retinopathy (DR) screening. However, their classification accuracy degrades as the input images distributionally shift from their training distribution. Further, even if the input is not a retina image, a standard DR classifier produces a high confident prediction that the image is `referable'. Our paper presents a Dirichlet Prior Network-based framework to address this issue. It utilizes an out-of-distribution (OOD) detector model and a DR classification model to improve generalizability by identifying OOD images. Experiments on real-world datasets indicate that the proposed framework can eliminate the unknown non-retina images and identify the distributionally shifted retina images for human intervention.
BibTeX
@inproceedings{nandy2021distributional,
  author = {Jay Nandy and Wynne Hsu and Mong-Li Lee},
  title = {{Distributional Shifts in Automated Diabetic Retinopathy Screening}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506635},
}
Figure