Non-local Attention Improves Description Generation for Retinal Images

Jia-Hong Huang, Ting-Wei Wu, Chao-Han Huck Yang, Zenglin Shi, I-Hung Lin, Jesper Tegnér, Marcel Worring

Abstract

Automated generation of medical reports from retinal images is a critical task that requires bridging the gap between visual data and complex clinical terminology. This paper presents TransFuser, a novel non-local attention mechanism that fuses retinal image features with expert-provided keywords. By treating the image as a "query" and keywords as "keys/values," the model captures intricate multi-modal interactions. Experimental results on the DeepEyeNet dataset demonstrate significant improvements over state-of-the-art baselines, with human evaluations confirming the clinical relevance of the generated reports.

Introduction

Generating accurate medical reports from retinal fundus images is a high-stakes challenge in medical computer vision. Unlike standard image captioning, medical descriptions often involve abstract clinical concepts that are not always explicitly visible in the raw pixels. Two primary hurdles exist: the information gap, where context beyond the image is required, and the fusion problem, where simple methods of combining image data with auxiliary information (like keywords) fail to capture nuanced relationships.

In clinical settings, ophthalmologists often record a small set of unordered keywords during an examination. This research proposes leveraging these keywords as semantic anchors to guide the report generation process, ensuring that the final output is both visually grounded and clinically accurate.

The TransFuser Architecture

The core contribution of this work is the TransFuser, a multi-modal feature fusion encoder inspired by the Transformer architecture. Unlike standard attention mechanisms that typically use the previously generated word as a query, TransFuser employs a unique "Image-as-Query" approach.

Overall architecture of the TransFuser model, featuring the encoder-decoder structure for medical report generation.

Image-as-Query and Non-local Attention

The model extracts image features using a CNN backbone to serve as the Query ($Q$). Simultaneously, expert keywords are embedded to form the Keys ($K$) and Values ($V$). By using scaled dot-product attention, the model calculates weights for each keyword based on its relevance to specific image regions. This non-local attention allows the model to "attend" to keywords that explain observed pathological features, effectively capturing mutual information between the two modalities.

Detailed view of the generation of Query (Q) from image features and Keys (K)/Values (V) from keyword embeddings.

Hybrid Feature Decoder

The resulting fused vector ($k_{final}$) is passed to a bidirectional LSTM decoder. At each time step, the decoder integrates three distinct inputs: global image features, the fused keyword-image vector, and the previously generated word. This hybrid approach ensures the model maintains a global context while focusing on the specific interactions identified by the TransFuser encoder.

Experimental Results

The model was evaluated primarily on the DeepEyeNet dataset, which contains 15,709 retinal images paired with expert keywords and full clinical descriptions.

Quantitative Performance

The TransFuser-reinforced model significantly outperformed several state-of-the-art baselines across standard NLP metrics:

  • BLEU-avg: +32% improvement over non-keyword baselines.
  • CIDEr: +2.5% improvement.
  • ROUGE: +25.4% improvement.

Ablation studies confirmed that the combination of "Image + Keywords" via TransFuser is superior to using either modality in isolation, proving that the model successfully learns to integrate disparate sources of information.

Clinical Validation

In a blind test, five retinal specialists evaluated the model-generated reports. The model achieved an average score of 3.6/5.0, compared to 4.5/5.0 for human-written reports. This indicates that the system produces descriptions that are clinically competitive and potentially useful for assisting practitioners.

Discussion and Limitations

While TransFuser represents a significant advancement, it has limitations. The model learns statistical correlations rather than a deep clinical understanding; for example, it might associate the keyword "degeneration" with certain visual patterns without understanding the underlying disease progression. Additionally, the model currently lacks the ability to incorporate patient metadata like age or gender, which are often crucial for comprehensive reports. Future work may involve "slot-filling" techniques to integrate these missing data points.

Conclusion

TransFuser demonstrates that non-local attention is a powerful tool for multi-modal medical report generation. By effectively fusing unordered expert keywords with spatial image data, the model bridges the semantic gap between pixels and prose, offering a promising path toward automated clinical documentation in ophthalmology.

Summary updated

Abstract

Automatically generating medical reports from retinal images is a difficult task in which an algorithm must generate semantically coherent descriptions for a given retinal image. Existing methods mainly rely on the input image to generate descriptions. However, many abstract medical concepts or descriptions cannot be generated based on image information only. In this work, we integrate additional information to help solve this task; we observe that early in the diagnosis process, ophthalmologists have usually written down a small set of keywords denoting important information. These keywords are then subsequently used to aid the later creation of medical reports for a patient. Since these keywords commonly exist and are useful for generating medical reports, we incorporate them into automatic report generation. Since we have two types of inputs expert-defined unordered keywords and images - effectively fusing features from these different modalities is challenging. To that end, we propose a new keyword-driven medical report generation method based on a non-local attention-based multi-modal feature fusion approach, TransFuser, which is capable of fusing features from different types of inputs based on such attention. Our experiments show the proposed method successfully captures the mutual information of keywords and image content. We further show our proposed keyword-driven generation model reinforced by the TransFuser is superior to baselines under the popular text evaluation metrics BLEU, CIDEr, and ROUGE. Trans-Fuser Github: https://github.com/Jhhuangkay/Non-local-Attention-ImprovesDescription-Generation-for-Retinal-Images.
BibTeX
@inproceedings{huang2022non,
  author = {Jia-Hong Huang and Ting-Wei Wu and Chao-Han Huck Yang and Zenglin Shi and I-Hung Lin and Jesper Tegnér and Marcel Worring},
  title = {{Non-local Attention Improves Description Generation for Retinal Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00331},
}
Figure