TL;DR
RetinaGuard is an "agentic" triage framework that uses a cost-aware approach to decide when an automated retinal screening decision can be trusted. By combining multiple specialized agents with Conformal Risk Control, it provides statistical safety guarantees while reducing computational overhead by only invoking expensive "second-opinion" tools when they are expected to provide significant value.
What Problem Does This Paper Address?
Deep learning has reached specialist-level performance in detecting retinal diseases, but "average accuracy" isn't enough for clinical screening programs. These programs need to know exactly which automated decisions are safe to accept and which must be referred to a human ophthalmologist.
The standard approach is to refer cases with low confidence. However, this doesn't account for image quality, doesn't provide statistical guarantees on the error rate of "accepted" cases, and doesn't consider the computational cost of gathering more evidence. RetinaGuard addresses these gaps by treating triage as an active evidence-gathering process.
Methods
The authors propose an agentic graph where a case passes through several stages, each adding more information:
- Quality Agent: Checks for photographic artifacts (blur, darkness, etc.) and refers "acquisition outliers" immediately.
- Diagnostic & Uncertainty Agents: Uses a primary model (RETFound) to make a prediction and estimates uncertainty via Monte Carlo dropout.
- Verification Agent: Invokes an independent "second opinion" model (ConvNeXt-B) to check for disagreements.
- Evidence Agent: Generates explainability maps (like Grad-CAM) to see if the model is looking at the right features.
A core innovation is the Value of Information (VoI) Gating. The system estimates whether the next agent in the sequence is likely to change the risk estimate enough to justify its computational cost (latency). If the expected risk reduction is low, the tool is skipped. Finally, the system uses Conformal Risk Control to set an acceptance threshold that guarantees the rate of "unsafe" automated decisions stays below a user-defined level (e.g., 5%).
Datasets
The study utilizes three public datasets:
- RFMiD: The primary dataset for training and internal testing, containing 3,200 images with 46 disease labels.
- ODIR-5K: Used for external validation to test how well the safety guarantees hold up under "distribution shift" (different populations/cameras).
- IDRiD: A small set with pixel-level lesion masks used specifically to validate whether the "Evidence Agent" actually points to real pathology.
Key Insights and Ideas
- The Utility of a "Second Opinion" is Relative: One of the most striking findings is that the benefit of an agentic approach depends on the strength of the primary model. If the primary model is weak, a verifier adds massive value. If the primary model is state-of-the-art, its own internal uncertainty estimates are often "aware" of its errors, making a second model redundant.
- Cost-Awareness in AI: Unlike most frameworks that run every available model, RetinaGuard's VoI gate treats computation as a finite resource. It successfully reduced computation by ~30% by skipping unnecessary verification steps.
- Quality as a Safety Net: The quality agent proved essential for robustness. While diagnostic models can sometimes be "confidently wrong" on blurry or dark images, the quality agent catches these before they can cause an unsafe automated acceptance.
Key Results
- Improved Triage: Using a weaker primary model, RetinaGuard caught 119 out of 280 unsafe decisions at a 20% referral budget, compared to only 88 for standard confidence-based referral.
- Certified Safety: At a nominal 5% unsafe level, RetinaGuard accepted 37.0% of cases automatically. In contrast, standard methods either accepted far fewer cases (10.3%) to maintain safety or failed to meet the safety target.
- Robustness to Corruption: Under severe image blur, the unsafe error rate for a standard model can skyrocket to over 70%. The quality agent successfully capped this risk by referring degraded images for re-acquisition.
Limitations and Future Work
The authors highlight a critical "reality check": conformal guarantees do not automatically transfer. When tested on the external ODIR-5K dataset, the error rate exceeded the 5% target. This suggests that "safety" is still highly dependent on the data distribution the model was calibrated on.
Future work aims to develop "distribution-robust" conformal methods and move toward clinically weighted losses, where missing a sight-threatening disease is penalized more heavily than missing a minor condition. They also plan to validate the quality agent against human "gradability" assessments.
Summary updated