DRG-Net: Interactive Joint Learning of Multi-lesion Segmentation and Classification for Diabetic Retinopathy Grading

Hasan Md Tusfiqur, Duy M. H. Nguyen, Mai T. N. Truong, Triet A. Nguyen, Binh T. Nguyen, Michael Barz, Hans-Juergen Profitlich, Ngoc T. T. Than, Ngan Le, Pengtao Xie, Daniel Sonntag

TL;DR

DRG-Net is a novel joint learning framework that simultaneously performs multi-lesion segmentation and disease grading for Diabetic Retinopathy (DR). By integrating unsupervised domain adaptation and an interactive human-in-the-loop module, the system allows clinicians to refine AI predictions through "weak" feedback (like bounding boxes), significantly improving performance and explainability across different clinical datasets.

What Problem Does This Paper Address?

Diabetic Retinopathy (DR) is a leading cause of blindness, yet accurate grading is time-consuming and requires specialized expertise. While deep learning has shown promise in automating this task, three major hurdles remain:

  1. Black-Box Nature: Most models provide a grade without explaining which lesions (e.g., hemorrhages or exudates) led to that decision.
  2. Data Scarcity and Domain Shift: High-quality, pixel-level lesion annotations are rare. Furthermore, a model trained on one hospital's equipment often fails when deployed in another due to differences in image distribution (domain shift).
  3. Disconnected Tasks: Lesion discovery and disease grading are clinically inseparable, yet most AI research treats them as independent problems.

Methods

The authors propose DRG-Net, which consists of two primary modules: the DRG-AI-System and the DRG-Expert-Interaction loop.

The DRG-AI-System

This module uses a multi-pronged approach to handle joint learning and domain adaptation:

  • Feature Generator (S-Net): A U-Net-based architecture that partitions retinal images into masks for four lesion types: Microaneurysms, Hemorrhages, Hard Exudates, and Soft Exudates.
  • Unsupervised Domain Adaptation (UDA): To handle domain shift, the authors employ three support networks:
  • PD-Net (Patch Discriminator): Improves segmentation accuracy on tiny lesions by forcing the model to distinguish between "real" and "fake" patches.
  • W-Net (Wasserstein Discriminator): Minimizes the Wasserstein distance between feature embeddings of the source and target domains.
  • AD-Net (Adversarial Domain Discriminator): Aligns the entropy distributions of the model’s predictions across domains, ensuring structural consistency.
  • Attention-Based Classifier: This module links lesions to grading. It uses low-level attention (integrating lesion maps into feature maps) and a novel high-level overlapping constraint. The latter forces the model's Class Activation Map (CAM)—the "heat map" showing where the model is looking—to overlap with the predicted lesion regions.
Fig. 2: A high-level overview of our proposed DRG-Net system that contains two main module of DRG-AI-System and DRG-Expert-Interaction. DRG-AI-System: Given a retinal image, our deep learning models will simultaneously generate three outcomes prediction (DR grade, lesion region, visual explanation). DRG-Expert-Interaction: By using an Intelligent User Interface (IUI), ophthalmologists can observe the system’s predictions and provide feedback to fine-tune the model.

The DRG-Expert-Interaction

This module enables an interactive machine learning (IML) loop. Experts can inspect the AI's predictions and provide feedback. Crucially, the system is robust to "noisy" feedback; experts can draw rough bounding boxes rather than precise pixel-wise masks to update and fine-tune the model.

Datasets

The framework was evaluated on three major benchmarks:

  • IDRID: Provides both high-resolution images for multi-lesion segmentation and 5-level severity grading labels.
  • FGADR: A large dataset containing both image-level grading and pixel-level lesion annotations.
  • EyePACS: A massive dataset (over 88,000 images) used primarily for evaluating the classification/grading performance.

Key Insights and Ideas

  • Joint Learning is Synergistic: Training the model to segment lesions actually helps it grade the disease more accurately, and vice versa.
  • Explainability via Correlation: By forcing the grading network to focus on segmented lesion areas, the resulting heatmaps become clinically meaningful. Clinicians can see exactly which lesions the AI used to justify a "Severe DR" diagnosis.
  • Domain Adaptation via Entropy: The use of AD-Net to align entropy distributions is a clever way to ensure that the model maintains high confidence in its predictions even when encountering images from a new, unlabeled dataset.
  • Robustness to Weak Supervision: The attention mechanism allows the model to filter out noise in user feedback, meaning clinicians don't need to spend hours on pixel-perfect annotations to improve the system.

Key Results

DRG-Net outperformed several state-of-the-art (SOTA) methods across multiple metrics:

  • Segmentation: On the IDRID dataset, the full DRG-Net (using TATL and PD-Net) improved the AUC-PR for hemorrhages by 9.4% compared to a standard S-Net baseline.
  • Domain Adaptation: In a cross-dataset scenario (IDRID $\rightarrow$ FGADR), the UDA components (AD-Net + W-Net) improved AUC-ROC and AUC-PR scores by approximately 12% and 15.3% respectively over non-adaptive baselines.
  • Grading Accuracy: The joint framework achieved a Quadratic Weighted Kappa of 0.911 on IDRID and 0.861 on FGADR, surpassing competitive models like CoLL and AFN-Net.
  • Human-in-the-Loop: Simulations showed that providing just 30% of user feedback data allowed the model to achieve performance comparable to fully supervised SOTA methods.

Limitations and Future Work

While comprehensive, the authors note a few areas for improvement:

  • Non-Differentiable Features: The current model doesn't explicitly account for geometric lesion characteristics (like area or radius) because these are difficult to formulate in a differentiable end-to-end learning strategy. The authors suggest exploring discrete optimization or geometric deep learning to bridge this gap.
  • Real-World UX: The user feedback was simulated. Future work involves building a production-ready intelligent user interface to test how real ophthalmologists interact with the system.
  • Continual Learning: As more user feedback is integrated, ensuring the model doesn't "forget" previous knowledge (catastrophic forgetting) remains an open research question.

Summary updated

Abstract

Diabetic Retinopathy (DR) is a leading cause of vision loss in the world, and early DR detection is necessary to prevent vision loss and support an appropriate treatment. In this work, we leverage interactive machine learning and introduce a joint learning framework, termed DRG-Net, to effectively learn both disease grading and multi-lesion segmentation. Our DRG-Net consists of two modules: (i) DRG-AI-System to classify DR Grading, localize lesion areas, and provide visual explanations; (ii) DRG-Expert-Interaction to receive feedback from user-expert and improve the DRG-AI-System. To deal with sparse data, we utilize transfer learning mechanisms to extract invariant feature representations by using Wasserstein distance and adversarial learning-based entropy minimization. Besides, we propose a novel attention strategy at both low- and high-level features to automatically select the most significant lesion information and provide explainable properties. In terms of human interaction, we further develop DRG-Net as a tool that enables expert users to correct the system's predictions, which may then be used to update the system as a whole. Moreover, thanks to the attention mechanism and loss functions constraint between lesion features and classification features, our approach can be robust given a certain level of noise in the feedback of users. We have benchmarked DRG-Net on the two largest DR datasets, i.e., IDRID and FGADR, and compared it to various state-of-the-art deep learning networks. In addition to outperforming other SOTA approaches, DRG-Net is effectively updated using user feedback, even in a weakly-supervised manner.
BibTeX
@misc{tusfiqur2022drg,
  author = {Hasan Md Tusfiqur and Duy M. H. Nguyen and Mai T. N. Truong and Triet A. Nguyen and Binh T. Nguyen and Michael Barz and Hans-Juergen Profitlich and Ngoc T. T. Than and Ngan Le and Pengtao Xie and Daniel Sonntag},
  title = {{DRG-Net: Interactive Joint Learning of Multi-lesion Segmentation and Classification for Diabetic Retinopathy Grading}},
  year = {2022},
  eprint = {2212.14615},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.14615},
}
Figure