Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"

Jia-Hong Huang, Ting-Wei Wu, Chao-Han Huck Yang, Marcel WorringarXiv 2021 · arXiv

Abstract

The increasing prevalence of retinal diseases like Diabetic Retinopathy (DR) and Age-related Macular Degeneration (AMD) has created a significant diagnostic workload for ophthalmologists. Automated medical report generation offers a solution, yet standard image captioning models often struggle to translate raw pixels into nuanced clinical descriptions. This paper introduces a Deep Context-Encoding Network that utilizes an LSTM-based encoder to fuse visual data with expert-defined keywords. By capturing the interactive relationship between images and clinical context, the proposed method achieves substantial improvements in captioning accuracy, outperforming previous state-of-the-art models by up to 16% in BLEU scores.

Introduction

Retinal diseases affect over 500 million people worldwide. While deep learning has revolutionized natural image captioning, medical reports present unique challenges. Unlike describing a "dog in a park," a medical report must synthesize abstract clinical concepts and specific pathological findings. Previous models typically relied on image features alone or used simplistic fusion methods (like averaging) to incorporate text context. These approaches often fail to capture the subtle, interactive relationship between a visual finding and its corresponding clinical keyword.

Methodology: Deep Context-Encoding

The authors propose a novel encoder-decoder architecture designed to extract "interactive information" from multi-modal inputs.

Multi-modal Input Encoder ($E_k$)

The core innovation lies in the LSTM-based encoder. Rather than simple concatenation, the retinal image—processed through a CNN feature extractor (e.g., VGG16, ResNet)—is used to "prime" the LSTM at the initial time step. Subsequently, a sequence of expert keywords is fed into the network. The final hidden state, $k_{final}$, represents a sophisticated, fused representation of the image as interpreted through the lens of the clinical keywords.

Fused-feature Decoder ($G_\theta$)

The decoder utilizes a bidirectional LSTM (BiLSTM) to generate the final report. At each generation step, the model integrates three distinct inputs: raw image features, the fused context feature ($k_{final}$), and the previously generated word. This "unrolling" process allows the model to learn how different keywords exert varying levels of influence on the final clinical description.

The proposed encoder-decoder architecture for context-aware retinal image captioning.

Experimental Setup

The model was evaluated using the DEN dataset, comprising:

  • 1,811 grey-scale Fluorescein Angiography (FA) images.
  • 13,898 colorful Color Fundus Photography (CFP) images.
  • Expert Annotations: Each image is paired with a sequence of input keywords and a ground-truth clinical description.

The vocabulary consists of approximately 4,000 to 4,300 words, with typical report lengths ranging between 5 and 10 words.

Results and Discussion

The proposed method consistently outperformed high-profile baselines, including models by Karpathy et al. and Vinyals et al. Quantitative improvements were significant across all metrics: BLEU-avg (+16%), CIDEr (+10.2%), and ROUGE (+8.6%).

Qualitative analysis reveals that the inclusion of expert keywords allows the model to identify specific pathologies that non-keyword-driven models miss. For instance, while baseline models might provide generic descriptions, the proposed network accurately identifies conditions such as "x-linked retinoschisis" and "dusn" (diffuse unilateral subacute neuroretinitis).

Qualitative comparison of generated captions showing the superior accuracy of the keyword-driven approach in identifying specific pathologies.

Conclusion and Ethical Considerations

The Deep Context-Encoding Network demonstrates that keywords act as a vital "lens," allowing models to re-interpret visual data more accurately. However, challenges remain. Standard NLP metrics like BLEU may not fully capture the clinical gravity of errors (e.g., misidentifying the "left" vs. "right" eye). Furthermore, the authors caution against the risk of "self-diagnosis" by patients. These tools are intended to assist specialists in managing high workloads, not to replace the critical oversight of a trained ophthalmologist. Future work will focus on incorporating broader patient metadata and refining evaluation metrics for medical accuracy.

Summary updated

Abstract

Automatically generating medical reports for retinal images is one of the promising ways to help ophthalmologists reduce their workload and improve work efficiency. In this work, we propose a new context-driven encoding network to automatically generate medical reports for retinal images. The proposed model is mainly composed of a multi-modal input encoder and a fused-feature decoder. Our experimental results show that our proposed method is capable of effectively leveraging the interactive information between the input image and context, i.e., keywords in our case. The proposed method creates more accurate and meaningful reports for retinal images than baseline models and achieves state-of-the-art performance. This performance is shown in several commonly used metrics for the medical report generation task: BLEU-avg (+16%), CIDEr (+10.2%), and ROUGE (+8.6%).
BibTeX
@misc{huang2021longer,
  author = {Jia-Hong Huang and Ting-Wei Wu and Chao-Han Huck Yang and Marcel Worring},
  title = {{Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"}},
  year = {2021},
  eprint = {2105.14538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.14538},
}
Figure