Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation

Mingjie Li, Wenjia Cai, Karin Verspoor, Shirui Pan, Xiaodan Liang, Xiaojun Chang

Introduction

Fundus Fluorescein Angiography (FFA) is a critical diagnostic procedure in ophthalmology, involving the capture of a temporal sequence of retinal images to identify vascular abnormalities. However, interpreting these sequences is a labor-intensive task requiring significant clinical expertise. While automated medical report generation has advanced, standard image-captioning models often struggle to bridge the gap between low-level visual features and high-level clinical interpretations. This paper introduces the Cross-modal Clinical Graph Transformer (CGT), a framework designed to integrate domain-specific clinical knowledge into the report generation process while mitigating the risks of "Knowledge Noise."

The Challenge of Knowledge Noise

Previous attempts to incorporate medical knowledge into AI models have frequently encountered two primary issues, collectively termed "Knowledge Noise" (KN):

  1. Misalignment: External knowledge bases like UMLS are often too broad, and their terminology may not align with the specific linguistic patterns and clinical context found in actual ophthalmic reports.
  2. Visual Diversion: When external information is injected without proper filtering, it can overshadow the actual visual evidence in the images. This leads to "hallucinations," where the model generates findings based on statistical likelihood in a database rather than what is actually visible in the scan.

Methodology: The CGT Framework

The CGT model addresses these challenges through a three-pronged architectural approach designed to ensure clinical relevance and visual fidelity.

In-Domain Knowledge Extraction

To solve the misalignment problem, the authors developed a custom NLP pipeline using an "Ai Parser" (based on SpaCy) for dependency parsing and entity linking. This system extracts clinical triples—consisting of subjects, relations, and objects (e.g., [hemorrhage, seen, macular])—directly from the training corpus. This ensures that the "knowledge" used by the model is perfectly aligned with the target report language.

Graph Restoration and Triple Restoration Loss

For new image sets, the model employs a Graph Restoration Module. Instead of mapping pixels directly to text, the model first attempts to "restore" a relevant clinical sub-graph from the visual data. A novel Triple Restoration Loss is introduced to supervise this process, penalizing the model if it fails to correctly identify the underlying clinical relationships. This provides a much stronger reasoning signal than standard word-matching objectives.

The architecture of the Cross-modal Clinical Graph Transformer (CGT), highlighting the Graph Restoration Module and the Triple Restoration Loss.

Cross-modal Encoder with Visible Matrix

To prevent visual diversion, the CGT utilizes a specialized "visible matrix" within its Transformer encoder. This matrix acts as a mask that restricts the flow of information, ensuring that clinical triples only interact with relevant visual features. By compressing dozens of FFA images into a single "compressed visual token," the model maintains a clear narrative focus driven by the clinical graph rather than redundant image features.

Experimental Results

The model was rigorously evaluated on the FFA-IR benchmark, the largest dataset of its kind, featuring over 1 million images and 10,790 reports.

Quantitative Performance

The CGT achieved state-of-the-art results across standard Natural Language Generation (NLG) metrics, including a significant 0.599 CIDEr score. More importantly, the impact of the Triple Restoration Loss was clearly demonstrated in the model's diagnostic accuracy. The area under the ROC curve (AUC) for identifying correct clinical triples improved from a near-random 0.55 to 0.78.

ROC curve demonstrating the significant improvement in clinical triple identification accuracy provided by the Triple Restoration Loss.

Clinical Validation

In a human study involving senior ophthalmologists, reports generated by the CGT were found to be clinically accurate 44.7% of the time, compared to only 21.6% for the baseline model. The explicit restoration of clinical triples also enhances the model's explainability, allowing clinicians to trace the AI's logic back to specific identified relationships.

Conclusion

The Cross-modal Clinical Graph Transformer represents a significant advancement in automated ophthalmic report generation. By prioritizing in-domain knowledge and implementing structural safeguards against noise, the model achieves both higher linguistic quality and greater clinical utility. Future work will focus on refining the automatic extraction process to reduce inherited biases and exploring the generalizability of the CGT framework to other medical imaging modalities.

Summary updated

BibTeX
@inproceedings{li2022cross,
  author = {Mingjie Li and Wenjia Cai and Karin Verspoor and Shirui Pan and Xiaodan Liang and Xiaojun Chang},
  title = {{Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.02000},
}
Figure