Abstract
Automated generation of medical reports from retinal images is a critical task that requires bridging the gap between visual data and complex clinical terminology. This paper presents TransFuser, a novel non-local attention mechanism that fuses retinal image features with expert-provided keywords. By treating the image as a "query" and keywords as "keys/values," the model captures intricate multi-modal interactions. Experimental results on the DeepEyeNet dataset demonstrate significant improvements over state-of-the-art baselines, with human evaluations confirming the clinical relevance of the generated reports.
Introduction
Generating accurate medical reports from retinal fundus images is a high-stakes challenge in medical computer vision. Unlike standard image captioning, medical descriptions often involve abstract clinical concepts that are not always explicitly visible in the raw pixels. Two primary hurdles exist: the information gap, where context beyond the image is required, and the fusion problem, where simple methods of combining image data with auxiliary information (like keywords) fail to capture nuanced relationships.
In clinical settings, ophthalmologists often record a small set of unordered keywords during an examination. This research proposes leveraging these keywords as semantic anchors to guide the report generation process, ensuring that the final output is both visually grounded and clinically accurate.
The TransFuser Architecture
The core contribution of this work is the TransFuser, a multi-modal feature fusion encoder inspired by the Transformer architecture. Unlike standard attention mechanisms that typically use the previously generated word as a query, TransFuser employs a unique "Image-as-Query" approach.
Image-as-Query and Non-local Attention
The model extracts image features using a CNN backbone to serve as the Query ($Q$). Simultaneously, expert keywords are embedded to form the Keys ($K$) and Values ($V$). By using scaled dot-product attention, the model calculates weights for each keyword based on its relevance to specific image regions. This non-local attention allows the model to "attend" to keywords that explain observed pathological features, effectively capturing mutual information between the two modalities.
Hybrid Feature Decoder
The resulting fused vector ($k_{final}$) is passed to a bidirectional LSTM decoder. At each time step, the decoder integrates three distinct inputs: global image features, the fused keyword-image vector, and the previously generated word. This hybrid approach ensures the model maintains a global context while focusing on the specific interactions identified by the TransFuser encoder.
Experimental Results
The model was evaluated primarily on the DeepEyeNet dataset, which contains 15,709 retinal images paired with expert keywords and full clinical descriptions.
Quantitative Performance
The TransFuser-reinforced model significantly outperformed several state-of-the-art baselines across standard NLP metrics:
- BLEU-avg: +32% improvement over non-keyword baselines.
- CIDEr: +2.5% improvement.
- ROUGE: +25.4% improvement.
Ablation studies confirmed that the combination of "Image + Keywords" via TransFuser is superior to using either modality in isolation, proving that the model successfully learns to integrate disparate sources of information.
Clinical Validation
In a blind test, five retinal specialists evaluated the model-generated reports. The model achieved an average score of 3.6/5.0, compared to 4.5/5.0 for human-written reports. This indicates that the system produces descriptions that are clinically competitive and potentially useful for assisting practitioners.
Discussion and Limitations
While TransFuser represents a significant advancement, it has limitations. The model learns statistical correlations rather than a deep clinical understanding; for example, it might associate the keyword "degeneration" with certain visual patterns without understanding the underlying disease progression. Additionally, the model currently lacks the ability to incorporate patient metadata like age or gender, which are often crucial for comprehensive reports. Future work may involve "slot-filling" techniques to integrate these missing data points.
Conclusion
TransFuser demonstrates that non-local attention is a powerful tool for multi-modal medical report generation. By effectively fusing unordered expert keywords with spatial image data, the model bridges the semantic gap between pixels and prose, offering a promising path toward automated clinical documentation in ophthalmology.
Summary updated