RetinaGuard: Risk-Controlled Agentic Triage for Multi-Disease Retinal Screening

Sohaib Sulman, Muhammad Atif Saeed, Akhtar Jamil, Alaa Ali Hameed, Roland AydinarXiv 2026 · arXiv

TL;DR

RetinaGuard is an "agentic" triage framework that uses a cost-aware approach to decide when an automated retinal screening decision can be trusted. By combining multiple specialized agents with Conformal Risk Control, it provides statistical safety guarantees while reducing computational overhead by only invoking expensive "second-opinion" tools when they are expected to provide significant value.

What Problem Does This Paper Address?

Deep learning has reached specialist-level performance in detecting retinal diseases, but "average accuracy" isn't enough for clinical screening programs. These programs need to know exactly which automated decisions are safe to accept and which must be referred to a human ophthalmologist.

The standard approach is to refer cases with low confidence. However, this doesn't account for image quality, doesn't provide statistical guarantees on the error rate of "accepted" cases, and doesn't consider the computational cost of gathering more evidence. RetinaGuard addresses these gaps by treating triage as an active evidence-gathering process.

Methods

The authors propose an agentic graph where a case passes through several stages, each adding more information:

  1. Quality Agent: Checks for photographic artifacts (blur, darkness, etc.) and refers "acquisition outliers" immediately.
  2. Diagnostic & Uncertainty Agents: Uses a primary model (RETFound) to make a prediction and estimates uncertainty via Monte Carlo dropout.
  3. Verification Agent: Invokes an independent "second opinion" model (ConvNeXt-B) to check for disagreements.
  4. Evidence Agent: Generates explainability maps (like Grad-CAM) to see if the model is looking at the right features.
The RetinaGuard agent graph, showing the flow from input to final decision.

A core innovation is the Value of Information (VoI) Gating. The system estimates whether the next agent in the sequence is likely to change the risk estimate enough to justify its computational cost (latency). If the expected risk reduction is low, the tool is skipped. Finally, the system uses Conformal Risk Control to set an acceptance threshold that guarantees the rate of "unsafe" automated decisions stays below a user-defined level (e.g., 5%).

Datasets

The study utilizes three public datasets:

  • RFMiD: The primary dataset for training and internal testing, containing 3,200 images with 46 disease labels.
  • ODIR-5K: Used for external validation to test how well the safety guarantees hold up under "distribution shift" (different populations/cameras).
  • IDRiD: A small set with pixel-level lesion masks used specifically to validate whether the "Evidence Agent" actually points to real pathology.
Fig. 1. Example images after field-of-view cropping: RFMiD test images with single labels, ODIR-5K images, and an IDRiD image with its lesion masks overlaid.

Key Insights and Ideas

  • The Utility of a "Second Opinion" is Relative: One of the most striking findings is that the benefit of an agentic approach depends on the strength of the primary model. If the primary model is weak, a verifier adds massive value. If the primary model is state-of-the-art, its own internal uncertainty estimates are often "aware" of its errors, making a second model redundant.
  • Cost-Awareness in AI: Unlike most frameworks that run every available model, RetinaGuard's VoI gate treats computation as a finite resource. It successfully reduced computation by ~30% by skipping unnecessary verification steps.
  • Quality as a Safety Net: The quality agent proved essential for robustness. While diagnostic models can sometimes be "confidently wrong" on blurry or dark images, the quality agent catches these before they can cause an unsafe automated acceptance.

Key Results

  • Improved Triage: Using a weaker primary model, RetinaGuard caught 119 out of 280 unsafe decisions at a 20% referral budget, compared to only 88 for standard confidence-based referral.
  • Certified Safety: At a nominal 5% unsafe level, RetinaGuard accepted 37.0% of cases automatically. In contrast, standard methods either accepted far fewer cases (10.3%) to maintain safety or failed to meet the safety target.
  • Robustness to Corruption: Under severe image blur, the unsafe error rate for a standard model can skyrocket to over 70%. The quality agent successfully capped this risk by referring degraded images for re-acquisition.
Performance under image corruption and distribution shift.

Limitations and Future Work

The authors highlight a critical "reality check": conformal guarantees do not automatically transfer. When tested on the external ODIR-5K dataset, the error rate exceeded the 5% target. This suggests that "safety" is still highly dependent on the data distribution the model was calibrated on.

Future work aims to develop "distribution-robust" conformal methods and move toward clinically weighted losses, where missing a sight-threatening disease is penalized more heavily than missing a minor condition. They also plan to validate the quality agent against human "gradability" assessments.

Summary updated

Abstract

Retinal screening programmes need to know which automated decisions can be accepted without review. This paper presents RetinaGuard, an agentic triage framework in which quality, diagnostic, uncertainty, verification and explanation agents gather evidence for each fundus photograph, a learned failure predictor estimates the probability that the decision is unsafe, auxiliary tools are invoked only when their expected value of information exceeds their cost, and the acceptance threshold is certified by conformal risk control. On the RFMiD test split, RetinaGuard was compared with confidence and calibrated-uncertainty referral at matched referral budgets and equal certified safety, using two RETFound primary models of different strength. With the weaker model (macro-AUROC 0.896), verifier disagreement added significant information, RetinaGuard captured 119 of 280 unsafe decisions at a 20% budget against 88 for lowest-confidence referral, and at a nominal 5% unsafe level it accepted 37.0% of cases against 10.3%. With the stronger model (0.925) the verifier added nothing beyond uncertainty and the advantage shrank to a few percentage points. In both settings its coverage at the nominal 5% level stayed near 37.7%, gating cut computation by about 30%, and a quality agent reduced unsafe acceptances under most image corruptions. The benefit of agentic evidence gathering thus depends on how much the primary model's own uncertainty already reveals about its errors.
BibTeX
@misc{sulman2026retinaguard,
  author = {Sohaib Sulman and Muhammad Atif Saeed and Akhtar Jamil and Alaa Ali Hameed and Roland Aydin},
  title = {{RetinaGuard: Risk-Controlled Agentic Triage for Multi-Disease Retinal Screening}},
  year = {2026},
  eprint = {2611.10689},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10689},
}
Figure