Abstract
The increasing prevalence of retinal diseases like Diabetic Retinopathy (DR) and Age-related Macular Degeneration (AMD) has created a significant diagnostic workload for ophthalmologists. Automated medical report generation offers a solution, yet standard image captioning models often struggle to translate raw pixels into nuanced clinical descriptions. This paper introduces a Deep Context-Encoding Network that utilizes an LSTM-based encoder to fuse visual data with expert-defined keywords. By capturing the interactive relationship between images and clinical context, the proposed method achieves substantial improvements in captioning accuracy, outperforming previous state-of-the-art models by up to 16% in BLEU scores.
Introduction
Retinal diseases affect over 500 million people worldwide. While deep learning has revolutionized natural image captioning, medical reports present unique challenges. Unlike describing a "dog in a park," a medical report must synthesize abstract clinical concepts and specific pathological findings. Previous models typically relied on image features alone or used simplistic fusion methods (like averaging) to incorporate text context. These approaches often fail to capture the subtle, interactive relationship between a visual finding and its corresponding clinical keyword.
Methodology: Deep Context-Encoding
The authors propose a novel encoder-decoder architecture designed to extract "interactive information" from multi-modal inputs.
Multi-modal Input Encoder ($E_k$)
The core innovation lies in the LSTM-based encoder. Rather than simple concatenation, the retinal image—processed through a CNN feature extractor (e.g., VGG16, ResNet)—is used to "prime" the LSTM at the initial time step. Subsequently, a sequence of expert keywords is fed into the network. The final hidden state, $k_{final}$, represents a sophisticated, fused representation of the image as interpreted through the lens of the clinical keywords.
Fused-feature Decoder ($G_\theta$)
The decoder utilizes a bidirectional LSTM (BiLSTM) to generate the final report. At each generation step, the model integrates three distinct inputs: raw image features, the fused context feature ($k_{final}$), and the previously generated word. This "unrolling" process allows the model to learn how different keywords exert varying levels of influence on the final clinical description.
Experimental Setup
The model was evaluated using the DEN dataset, comprising:
- 1,811 grey-scale Fluorescein Angiography (FA) images.
- 13,898 colorful Color Fundus Photography (CFP) images.
- Expert Annotations: Each image is paired with a sequence of input keywords and a ground-truth clinical description.
The vocabulary consists of approximately 4,000 to 4,300 words, with typical report lengths ranging between 5 and 10 words.
Results and Discussion
The proposed method consistently outperformed high-profile baselines, including models by Karpathy et al. and Vinyals et al. Quantitative improvements were significant across all metrics: BLEU-avg (+16%), CIDEr (+10.2%), and ROUGE (+8.6%).
Qualitative analysis reveals that the inclusion of expert keywords allows the model to identify specific pathologies that non-keyword-driven models miss. For instance, while baseline models might provide generic descriptions, the proposed network accurately identifies conditions such as "x-linked retinoschisis" and "dusn" (diffuse unilateral subacute neuroretinitis).
Conclusion and Ethical Considerations
The Deep Context-Encoding Network demonstrates that keywords act as a vital "lens," allowing models to re-interpret visual data more accurately. However, challenges remain. Standard NLP metrics like BLEU may not fully capture the clinical gravity of errors (e.g., misidentifying the "left" vs. "right" eye). Furthermore, the authors caution against the risk of "self-diagnosis" by patients. These tools are intended to assist specialists in managing high workloads, not to replace the critical oversight of a trained ophthalmologist. Future work will focus on incorporating broader patient metadata and refining evaluation metrics for medical accuracy.
Summary updated