Introduction
Fundus Fluorescein Angiography (FFA) is a critical diagnostic procedure in ophthalmology, involving the capture of a temporal sequence of retinal images to identify vascular abnormalities. However, interpreting these sequences is a labor-intensive task requiring significant clinical expertise. While automated medical report generation has advanced, standard image-captioning models often struggle to bridge the gap between low-level visual features and high-level clinical interpretations. This paper introduces the Cross-modal Clinical Graph Transformer (CGT), a framework designed to integrate domain-specific clinical knowledge into the report generation process while mitigating the risks of "Knowledge Noise."
The Challenge of Knowledge Noise
Previous attempts to incorporate medical knowledge into AI models have frequently encountered two primary issues, collectively termed "Knowledge Noise" (KN):
- Misalignment: External knowledge bases like UMLS are often too broad, and their terminology may not align with the specific linguistic patterns and clinical context found in actual ophthalmic reports.
- Visual Diversion: When external information is injected without proper filtering, it can overshadow the actual visual evidence in the images. This leads to "hallucinations," where the model generates findings based on statistical likelihood in a database rather than what is actually visible in the scan.
Methodology: The CGT Framework
The CGT model addresses these challenges through a three-pronged architectural approach designed to ensure clinical relevance and visual fidelity.
In-Domain Knowledge Extraction
To solve the misalignment problem, the authors developed a custom NLP pipeline using an "Ai Parser" (based on SpaCy) for dependency parsing and entity linking. This system extracts clinical triples—consisting of subjects, relations, and objects (e.g., [hemorrhage, seen, macular])—directly from the training corpus. This ensures that the "knowledge" used by the model is perfectly aligned with the target report language.
Graph Restoration and Triple Restoration Loss
For new image sets, the model employs a Graph Restoration Module. Instead of mapping pixels directly to text, the model first attempts to "restore" a relevant clinical sub-graph from the visual data. A novel Triple Restoration Loss is introduced to supervise this process, penalizing the model if it fails to correctly identify the underlying clinical relationships. This provides a much stronger reasoning signal than standard word-matching objectives.
Cross-modal Encoder with Visible Matrix
To prevent visual diversion, the CGT utilizes a specialized "visible matrix" within its Transformer encoder. This matrix acts as a mask that restricts the flow of information, ensuring that clinical triples only interact with relevant visual features. By compressing dozens of FFA images into a single "compressed visual token," the model maintains a clear narrative focus driven by the clinical graph rather than redundant image features.
Experimental Results
The model was rigorously evaluated on the FFA-IR benchmark, the largest dataset of its kind, featuring over 1 million images and 10,790 reports.
Quantitative Performance
The CGT achieved state-of-the-art results across standard Natural Language Generation (NLG) metrics, including a significant 0.599 CIDEr score. More importantly, the impact of the Triple Restoration Loss was clearly demonstrated in the model's diagnostic accuracy. The area under the ROC curve (AUC) for identifying correct clinical triples improved from a near-random 0.55 to 0.78.
Clinical Validation
In a human study involving senior ophthalmologists, reports generated by the CGT were found to be clinically accurate 44.7% of the time, compared to only 21.6% for the baseline model. The explicit restoration of clinical triples also enhances the model's explainability, allowing clinicians to trace the AI's logic back to specific identified relationships.
Conclusion
The Cross-modal Clinical Graph Transformer represents a significant advancement in automated ophthalmic report generation. By prioritizing in-domain knowledge and implementing structural safeguards against noise, the model achieves both higher linguistic quality and greater clinical utility. Future work will focus on refining the automatic extraction process to reduce inherited biases and exploring the generalizability of the CGT framework to other medical imaging modalities.
Summary updated