DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation

Jia-Hong Huang, Chao-Han Huck Yang, Fangyu Liu, Meng Tian, Yi-Chieh Liu, Ting-Wei Wu, I-Hung Lin, Kang Wang, Hiromasa Morikawa, Hernghua Chang, Jesper Tegnér, Marcel Worring

Abstract

Automating the generation of medical reports from retinal images represents a significant step toward efficient ophthalmological diagnostics. This paper introduces DeepOpht, a framework that integrates disease identification with natural language clinical descriptions. By leveraging a novel, large-scale dataset (DeepEyeNet) and a keyword-driven generation approach, DeepOpht produces structured medical reports accompanied by visual explanations. Experimental results demonstrate that this multi-modal approach significantly improves the accuracy and clinical relevance of automated reports compared to standard image-to-text baselines.

Introduction

Retinal diseases, including Age-related Macular Degeneration (AMD) and Diabetic Retinopathy (DR), are leading causes of vision loss worldwide. While advances in deep learning have enabled high-accuracy classification of these conditions, the clinical workflow remains bottlenecked by the manual effort required to draft detailed medical reports. Specialists must meticulously analyze fundus images to describe pathological features, a process that is both time-consuming and subject to inter-observer variability.

Traditional diagnostic procedures rely heavily on manual interpretation and documentation. The proposed multimedia-based diagnosis framework aims to streamline this by introducing automated point-of-care modules that assist doctors in generating comprehensive reports.

Comparison between traditional diagnosis procedures and the proposed multimedia-based framework.

The DeepOpht Framework

The DeepOpht system is designed to bridge the gap between visual data and clinical text through three integrated modules:

  1. Retinal Disease Identifier (RDI): Utilizing deep CNN architectures such as ResNet and VGG, this module classifies images into one of 265 unique retinal symptoms or disease categories.
  2. Clinical Description Generator (CDG): This module employs an encoder-decoder (CNN + LSTM) architecture. To ensure clinical precision, it uses a "keyword-driven" method where specific pathological keywords are extracted as an intermediate step before full sentence generation.
  3. Visual Explanation Module: To address the "black-box" nature of AI, the system uses Class Activation Mapping (CAM) to generate heatmaps. These highlight the retinal regions that influenced the model's decision, allowing for clinician verification.
Architecture of the DeepOpht framework for automated medical report generation.

The final output is a structured, table-based medical report that combines the identified disease, relevant keywords, a detailed clinical description, and visual evidence.

The DeepEyeNet Dataset

A core contribution of this work is the introduction of DeepEyeNet (DEN), a large-scale dataset specifically designed for medical report generation. DEN comprises 15,709 images, surpassing many existing retinal datasets in both scale and metadata richness. Each entry includes disease labels, specific keywords, and full-length clinical descriptions across diverse imaging modalities like Color Fundus Photography (CFP) and Fluorescein Angiography (FA).

To validate the visual explanation module, the authors also developed a gold-standard set of 300 images manually annotated and sketched by professional ophthalmologists to ensure the AI's "attention" aligns with actual clinical pathology.

Key Results and Insights

The DeepOpht framework was evaluated on its ability to both classify diseases and generate accurate text:

  • Classification Performance: The VGG19-based RDI achieved a Prec@1 of 54.23% and a Prec@5 of 80.75% across 265 classes. Given the fine-grained nature of the task, these results indicate a robust ability to distinguish between subtle pathological variations.
  • Report Accuracy: The keyword-driven approach proved superior to standard image captioning models. On the CIDEr metric, the model achieved a score of 0.361, more than doubling the performance of the baseline (0.167).
  • Transfer Learning: The authors found that using ImageNet-pre-trained weights significantly boosted performance, suggesting that low-level feature detection (edges, textures) transfers effectively to the medical domain.

Conclusion and Future Work

DeepOpht demonstrates the feasibility of automating complex medical reporting for retinal diseases. By anchoring text generation in clinical keywords and providing visual justifications, the framework enhances both the accuracy and the interpretability of AI-driven diagnostics. Future research will focus on mitigating demographic hallucinations and improving performance on rare disease classes through more sophisticated data balancing and feature fusion techniques.

Summary updated

Abstract

In this work, we propose an AI-based method that intends to improve the conventional retinal disease treatment procedure and help ophthalmologists increase diagnosis efficiency and accuracy. The proposed method is composed of a deep neural networks-based (DNN-based) module, including a retinal disease identifier and clinical description generator, and a DNN visual explanation module. To train and validate the effectiveness of our DNN-based module, we propose a large-scale retinal disease image dataset. Also, as ground truth, we provide a retinal image dataset manually labeled by ophthalmologists to qualitatively show, the proposed AI-based method is effective. With our experimental results, we show that the proposed method is quantitatively and qualitatively effective. Our method is capable of creating meaningful retinal image descriptions and visual explanations that are clinically relevant.
BibTeX
@inproceedings{huang2021deepopht,
  author = {Jia-Hong Huang and Chao-Han Huck Yang and Fangyu Liu and Meng Tian and Yi-Chieh Liu and Ting-Wei Wu and I-Hung Lin and Kang Wang and Hiromasa Morikawa and Hernghua Chang and Jesper Tegnér and Marcel Worring},
  title = {{DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00249},
}
Figure