Deep Context-Encoding Network For Retinal Image Captioning

Jia-Hong Huang, Ting-Wei Wu, Chao-Han Huck Yang, Marcel Worring

TL;DR

This paper introduces a novel deep learning model for automatically generating medical reports from retinal images. By effectively capturing the interplay between image features and expert-provided keywords, their "Deep Context-Encoding Network" significantly improves the accuracy and meaningfulness of generated reports, achieving state-of-the-art performance on key evaluation metrics.

What Problem Does This Paper Address?

Ophthalmologists face an overwhelming workload due to the increasing prevalence of retinal diseases. Automating the generation of medical reports for retinal images is a promising way to alleviate this burden and improve efficiency. While traditional image captioning models can describe natural images, they struggle with the abstract medical concepts and detailed descriptions required for medical reports. Previous attempts at context-driven medical report generation, which incorporate keywords, often use simple fusion methods (like averaging) that fail to fully leverage the rich interactions between image content and contextual keywords. This paper aims to bridge this gap by proposing a more sophisticated method for integrating image and keyword information to produce more accurate and clinically relevant reports.

Methods

The core of the proposed solution is a new end-to-end encoder-decoder architecture. The multi-modal input encoder is designed to capture the interactive information between the retinal image and the input keywords. Unlike previous methods that might simply average features, this encoder uses a Long Short-Term Memory (LSTM) based structure. The image is introduced as an initial input to the LSTM, followed by the embedded keyword vectors. This allows the model to learn how each keyword interacts with the image features over time. The output of this encoder is a fused vector that encapsulates this rich, interactive information.

This fused vector is then fed into a fused-feature decoder, which is a bidirectional LSTM. This decoder generates the medical report word by word, conditioned on the image features and the fused context vector. The authors emphasize that this approach allows for a more nuanced understanding of how keywords influence the interpretation of the retinal image, leading to more accurate and contextually appropriate report generation.

Datasets

The study utilizes a large-scale retinal image dataset introduced by the authors of [1]. This dataset comprises:

  • 1,811 grayscale Fluorescein Angiography (FA) images.
  • 13,898 colorful Color Fundus Photography (CFP) images.

Each image in the dataset is associated with two types of labels:

  • A clinical description (used as the ground truth for report generation).
  • Expert-defined keywords (used as contextual input for the model).

The dataset is split into training, validation, and testing sets with a 60%/20%/20% distribution, respectively.

Key Insights and Ideas

  • Beyond Simple Fusion: The paper strongly argues that simply averaging or multiplying image and keyword features is insufficient. The true value lies in capturing the interactive information, where keywords dynamically influence the interpretation of image regions and vice-versa.
  • Sequential Context Matters: Using an LSTM to process keywords sequentially after the image allows the model to build a richer understanding of the context. This sequential processing is key to learning the nuanced relationships between different keywords and the image.
  • Expert Keywords are Crucial: The authors highlight the advantage of using expert-defined keywords over model-generated tags. Expert keywords provide reliable, unbiased information that significantly aids the model's learning process.
  • Domain-Specific Challenges: The paper acknowledges that standard image captioning models don't generalize well to medical images, necessitating specialized approaches like context-driven methods.

Key Results

The proposed "Deep Context-Encoding Network" demonstrates significant improvements over baseline methods:

  • State-of-the-Art Performance: The model achieves state-of-the-art results on standard medical report generation metrics.
  • Quantitative Gains: Compared to the baseline method [1] (which also uses keywords but with a simpler fusion), the proposed method shows substantial improvements:
  • BLEU-avg: +16%
    • CIDEr: +10.2%
    • ROUGE: +8.6%
  • Qualitative Improvements: The generated captions are described as more accurate and meaningful, effectively leveraging keywords to describe important characteristics of retinal images. While the model cannot infer information not present in the image (like age or gender), it excels at generating clinically relevant descriptions based on the provided image and keywords.

Limitations and Future Work

  • Automatic Evaluation Metrics: The authors note that while their model performs well according to automatic metrics (BLEU, CIDEr, ROUGE), these metrics are not perfectly correlated with human judgment for complex medical descriptions. They suggest that developing better evaluation metrics for medical image captioning remains an open challenge.
  • Inferred Information: The model, like its predecessors, cannot infer information not directly present in the image (e.g., age, gender, skin color). The authors suggest that such information would ideally be part of the dataset or handled through post-processing.
  • Ethical Considerations: The paper touches upon the broader societal impact, including the potential for misuse by untrained individuals, leading to misdiagnosis or delayed treatment. They encourage further research into the ethical use and limitations of such automated systems.
  • Future Directions: The work naturally leads to exploring more advanced fusion mechanisms and potentially incorporating other forms of contextual information beyond keywords. The authors also implicitly suggest the need for larger, more diverse medical imaging datasets with richer annotations.

Summary updated

Abstract

Automatically generating medical reports for retinal images is one of the promising ways to help ophthalmologists reduce their workload and improve work efficiency. In this work, we propose a new context-driven encoding network to automatically generate medical reports for retinal images. The proposed model is mainly composed of a multi-modal input encoder and a fused-feature decoder. Our experimental results show that our proposed method is capable of effectively leveraging the interactive information between the input image and context, i.e., keywords in our case. The proposed method creates more accurate and meaningful reports for retinal images than baseline models and achieves state-of-the-art performance. This performance is shown in several commonly used metrics for the medical report generation task: BLEUavg (+16%), CIDEr (+10.2%), and ROUGE (+8.6%).
BibTeX
@inproceedings{huang2021deep,
  author = {Jia-Hong Huang and Ting-Wei Wu and Chao-Han Huck Yang and Marcel Worring},
  title = {{Deep Context-Encoding Network For Retinal Image Captioning}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506803},
}

Loading related content...

Figure