Abstract
Automating the generation of medical reports from retinal images represents a significant step toward efficient ophthalmological diagnostics. This paper introduces DeepOpht, a framework that integrates disease identification with natural language clinical descriptions. By leveraging a novel, large-scale dataset (DeepEyeNet) and a keyword-driven generation approach, DeepOpht produces structured medical reports accompanied by visual explanations. Experimental results demonstrate that this multi-modal approach significantly improves the accuracy and clinical relevance of automated reports compared to standard image-to-text baselines.
Introduction
Retinal diseases, including Age-related Macular Degeneration (AMD) and Diabetic Retinopathy (DR), are leading causes of vision loss worldwide. While advances in deep learning have enabled high-accuracy classification of these conditions, the clinical workflow remains bottlenecked by the manual effort required to draft detailed medical reports. Specialists must meticulously analyze fundus images to describe pathological features, a process that is both time-consuming and subject to inter-observer variability.
Traditional diagnostic procedures rely heavily on manual interpretation and documentation. The proposed multimedia-based diagnosis framework aims to streamline this by introducing automated point-of-care modules that assist doctors in generating comprehensive reports.
The DeepOpht Framework
The DeepOpht system is designed to bridge the gap between visual data and clinical text through three integrated modules:
- Retinal Disease Identifier (RDI): Utilizing deep CNN architectures such as ResNet and VGG, this module classifies images into one of 265 unique retinal symptoms or disease categories.
- Clinical Description Generator (CDG): This module employs an encoder-decoder (CNN + LSTM) architecture. To ensure clinical precision, it uses a "keyword-driven" method where specific pathological keywords are extracted as an intermediate step before full sentence generation.
- Visual Explanation Module: To address the "black-box" nature of AI, the system uses Class Activation Mapping (CAM) to generate heatmaps. These highlight the retinal regions that influenced the model's decision, allowing for clinician verification.
The final output is a structured, table-based medical report that combines the identified disease, relevant keywords, a detailed clinical description, and visual evidence.
The DeepEyeNet Dataset
A core contribution of this work is the introduction of DeepEyeNet (DEN), a large-scale dataset specifically designed for medical report generation. DEN comprises 15,709 images, surpassing many existing retinal datasets in both scale and metadata richness. Each entry includes disease labels, specific keywords, and full-length clinical descriptions across diverse imaging modalities like Color Fundus Photography (CFP) and Fluorescein Angiography (FA).
To validate the visual explanation module, the authors also developed a gold-standard set of 300 images manually annotated and sketched by professional ophthalmologists to ensure the AI's "attention" aligns with actual clinical pathology.
Key Results and Insights
The DeepOpht framework was evaluated on its ability to both classify diseases and generate accurate text:
- Classification Performance: The VGG19-based RDI achieved a Prec@1 of 54.23% and a Prec@5 of 80.75% across 265 classes. Given the fine-grained nature of the task, these results indicate a robust ability to distinguish between subtle pathological variations.
- Report Accuracy: The keyword-driven approach proved superior to standard image captioning models. On the CIDEr metric, the model achieved a score of 0.361, more than doubling the performance of the baseline (0.167).
- Transfer Learning: The authors found that using ImageNet-pre-trained weights significantly boosted performance, suggesting that low-level feature detection (edges, textures) transfers effectively to the medical domain.
Conclusion and Future Work
DeepOpht demonstrates the feasibility of automating complex medical reporting for retinal diseases. By anchoring text generation in clinical keywords and providing visual justifications, the framework enhances both the accuracy and the interpretability of AI-driven diagnostics. Future research will focus on mitigating demographic hallucinations and improving performance on rare disease classes through more sophisticated data balancing and feature fusion techniques.
Summary updated