DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation

Jia-Hong Huang, Chao-Han Huck Yang, Fangyu Liu, Meng Tian, Yi-Chieh Liu, Ting-Wei Wu, I-Hung Lin, Kang Wang, Hiromasa Morikawa, Hernghua Chang, Jesper Tegnér, Marcel Worring

TL;DR

This paper introduces DeepOpht, an AI system designed to automate the generation of medical reports for retinal images. By combining a deep learning model for disease identification and clinical description with a visual explanation module, DeepOpht aims to improve diagnostic efficiency and accuracy for ophthalmologists. The authors also present a new, large-scale dataset, DeepEyeNet (DEN), to train and validate their approach.

What Problem Does This Paper Address?

The diagnosis of retinal diseases and the subsequent creation of medical reports are time-consuming processes for ophthalmologists. With the increasing prevalence of conditions like Age-related Macular Degeneration (AMD) and Diabetic Retinopathy (DR), the workload on these specialists is expected to grow significantly. Current AI advancements, particularly in deep learning for computer vision and natural language processing, offer a promising avenue to alleviate this burden. This paper addresses the need for an AI system that can not only identify retinal diseases but also generate comprehensive and clinically relevant medical reports, thereby enhancing diagnostic efficiency and accuracy.

Methods

DeepOpht is built upon two core components: a DNN-based module and a DNN visual explanation module.

The DNN-based module itself comprises two sub-modules:

  1. Retinal Disease Identifier (RDI): This component uses deep learning models (specifically, architectures from [18, 53]) pre-trained on ImageNet and then fine-tuned on the proposed DEN dataset. The rationale for pre-training is to leverage learned low-level features, which are crucial for analyzing color-rich retinal images.
  2. Clinical Description Generator (CDG): This sub-module generates textual descriptions of the retinal images. It employs a CNN-based encoder (e.g., MobileNetV2, VGG16) to extract image features and an LSTM (a type of RNN) as a decoder to generate the description word by word. A key innovation here is the incorporation of a keyword-driven method to reinforce the CDG. This involves using a keyword embedding module to encode relevant terms, which are then fused with image features to improve the generated descriptions. Beam-search is used to optimize the output.

The DNN visual explanation module utilizes techniques like Class Activation Mapping (CAM) to visualize which parts of the retinal image are most influential in the model's predictions. This helps in understanding the model's decision-making process and verifying that its focus aligns with clinically recognized features.

Datasets

The authors introduce a new, large-scale dataset called DeepEyeNet (DEN).

  • DEN Dataset: Contains 15,709 retinal images, comprising 1,811 grayscale FA (Fluorescein Angiography) images and 13,898 color CFP (Color Fundus Photography) images. It is unique in providing three types of labels for each image: the name of the disease, keywords, and a clinical description, all defined by ophthalmologists. The dataset is split into 60% training, 20% validation, and 20% testing sets.
  • Additional Dataset: A smaller dataset of 300 retinal images, manually labeled by ophthalmologists, is used for qualitative evaluation of the visual explanation module.

The paper also provides a comprehensive comparison of DEN with existing retinal image datasets, highlighting DEN's larger size and richer labeling.

Key Insights and Ideas

  • Keyword-Reinforced Captioning: The integration of keywords significantly improves the quality of generated clinical descriptions, suggesting that leveraging domain-specific terminology is crucial for medical report generation.
  • Visual Explanations for Trust: Using CAM to visualize model attention maps and comparing them to ophthalmologists' knowledge provides a critical bridge for building trust in AI-driven diagnostic tools. It demonstrates that the AI is "looking" at the right things.
  • The Challenge of Medical Text: Generating accurate medical captions is inherently more difficult than natural image captioning due to the longer descriptions, more abstract vocabulary, and specific domain knowledge required.
  • Pre-training is Key: Pre-training deep learning models on large, general datasets like ImageNet is essential for achieving good performance on specialized medical imaging tasks, especially when the medical dataset itself is limited in size.

Key Results

  • RDI Performance: The RDI model based on [53] with ImageNet pre-training achieved the best performance, outperforming models without pre-training and other architectures. While Prec@1 accuracy was challenging due to the large number of classes (265) and limited training data, Prec@5 was more indicative of real-world utility.
  • CDG Performance: Keyword-driven CDG models consistently outperformed their non-keyword-driven counterparts across various metrics (BLEU-1, BLEU-2, BLEU-3, BLEU-4, BLEU-avg, CIDEr, ROUGE). The model based on [29] with keywords showed the best overall performance.
  • Qualitative Evaluation: The visual explanations generated by CAM were shown to align with clinically recognized lesion areas, demonstrating the model's ability to highlight relevant features. Generated clinical descriptions, while not always perfect with age/gender, captured important characteristics of retinal images.

Limitations and Future Work

  • Data Size for Deep Models: The authors note that even with a large dataset like DEN, the number of training images might still be insufficient for extremely deep models, especially given the 265 distinct retinal disease classes.
  • Accuracy of Age and Gender: The generated clinical descriptions sometimes struggle to accurately predict "age" and "gender," which are noted as difficult for automatic algorithms.
  • Subjectivity of Medical Text Evaluation: The paper acknowledges that evaluating image description generators is subjective, and standard metrics may not fully capture the quality of medical captions.
  • Future Work: The paper implicitly suggests further research into more sophisticated fusion methods for image and text features, exploring more advanced deep learning architectures, and potentially expanding the dataset with more diverse and complex cases. The visual explanation module's role in improving diagnostic accuracy is also a promising avenue for future investigation.

Summary updated

Abstract

In this work, we propose an AI-based method that intends to improve the conventional retinal disease treatment procedure and help ophthalmologists increase diagnosis efficiency and accuracy. The proposed method is composed of a deep neural networks-based (DNN-based) module, including a retinal disease identifier and clinical description generator, and a DNN visual explanation module. To train and validate the effectiveness of our DNN-based module, we propose a large-scale retinal disease image dataset. Also, as ground truth, we provide a retinal image dataset manually labeled by ophthalmologists to qualitatively show the proposed AI-based method is effective. With our experimental results, we show that the proposed method is quantitatively and qualitatively effective. Our method is capable of creating meaningful retinal image descriptions and visual explanations that are clinically relevant.https://github.com/Jhhuangkay/DeepOpht-Medical-Report-Generation-for-Retinal-Images-via-Deep-Models-and-Visual-Explanation.
BibTeX
@inproceedings{huang2021deepopht,
  author = {Jia-Hong Huang and Chao-Han Huck Yang and Fangyu Liu and Meng Tian and Yi-Chieh Liu and Ting-Wei Wu and I-Hung Lin and Kang Wang and Hiromasa Morikawa and Hernghua Chang and Jesper Tegnér and Marcel Worring},
  title = {{DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00249},
}
Figure