MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training for X-ray Diagnosis

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie

Abstract

Medical vision-language pre-training (VLP) typically relies on matching raw radiology reports with images, a process hindered by linguistic noise and the need for expert domain knowledge. We present MedKLIP, a novel framework that extracts structured clinical triplets—comprising entity, location, and existence—and enriches them with professional medical definitions. This approach allows the model to learn fine-grained visual patterns associated with specific diseases. MedKLIP achieves state-of-the-art performance in zero-shot diagnosis, spatial grounding, and low-data transfer across multiple chest X-ray benchmarks.

Introduction

In general computer vision, models like CLIP benefit from millions of web-scraped image-caption pairs. However, the medical domain faces three significant hurdles:

  1. Data Scarcity: Paired medical datasets are orders of magnitude smaller than general-purpose datasets.
  2. Linguistic Complexity: Radiology reports are free-form, containing complex grammar, negations, and stylistic variations that obscure core clinical findings.
  3. Expertise Gap: Medical concepts are fine-grained; a model must understand the underlying visual attributes of a condition, such as the specific opacities associated with pneumonia.

Existing VLP methods often attempt "coarse-grained" matching of entire reports to images, ignoring the structured domain knowledge available in medical textbooks. MedKLIP addresses these issues by leveraging structured supervision and external medical knowledge.

Methods

MedKLIP introduces three primary innovations to the VLP pipeline:

1. Triplet Extraction

Instead of processing raw text, MedKLIP utilizes tools like RadGraph to convert reports into structured triplets: {entity, position, existence}. For instance, the phrase "Increased right lower lobe opacity" is distilled into {Opacity, Right lower lobe, True}. This removes grammatical noise and focuses the model on clinical facts.

2. Knowledge-enhanced Encoding

To bridge the expertise gap, the model queries medical knowledge bases (e.g., UMLS, Wikipedia) for detailed descriptions of clinical entities. These definitions decompose complex diseases into basic visual attributes, enabling the model to recognize "unseen" diseases by identifying shared attributes—a key driver for zero-shot learning.

3. Transformer-based Fusion

MedKLIP employs a Transformer Decoder where medical entities serve as Queries that attend to image patches (Keys/Values). This architecture naturally aligns specific medical terms with spatial regions in the X-ray, providing inherent explainability through attention maps.

Overview of the MedKLIP framework, including report pre-processing, knowledge-enhanced encoding, and the training flow.

Evaluation and Results

The model was pre-trained on the MIMIC-CXR v2 dataset (227k studies) and evaluated on several public benchmarks, including ChestX-ray14, RSNA Pneumonia, and SIIM-ACR Pneumothorax.

Zero-Shot Performance

MedKLIP significantly outperforms existing state-of-the-art (SOTA) models like CheXzero and BioViL in zero-shot classification. For example, on the SIIM-ACR Pneumothorax dataset, MedKLIP achieved an AUC of 0.8924, compared to CheXzero’s 0.6879.

Quantitative comparison of MedKLIP against SOTA methods across three major chest X-ray datasets.

Grounding and Explainability

The entity-level query mechanism allows MedKLIP to produce accurate heatmaps for disease localization. In the "Pointing Game" metric on the RSNA Pneumonia dataset, MedKLIP scored 0.8721, surpassing GLoRIA's 0.7607.

Robustness and Transfer Learning

MedKLIP demonstrates exceptional data efficiency. When fine-tuning with only 1% of available data, it maintains high performance, indicating that the pre-trained representations are robust and clinically relevant. Furthermore, it shows strong generalization to "unseen" conditions like COVID-19 by leveraging attribute-based definitions.

Conclusion

MedKLIP represents a shift in medical VLP from raw text matching to knowledge-driven, structured supervision. By incorporating medical definitions and clinical triplets, the framework achieves superior generalization and explainability. Future work may address the dependency on initial triplet extraction tools and extend the approach to 3D modalities like CT or MRI.

Summary updated

Abstract

In this paper, we consider enhancing medical visual-language pre-training (VLP) with domain-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice. In particular, we make the following contributions: First, unlike existing works that directly process the raw reports, we adopt a novel triplet extraction module to extract the medical-related information, avoiding unnecessary complexity from language grammar and enhancing the supervision signals; Second, we propose a novel triplet encoding module with entity translation by querying a knowledge base, to exploit the rich domain knowledge in medical field, and implicitly build relationships between medical entities in the language embedding space; Third, we propose to use a Transformer-based fusion model for spatially aligning the entity description with visual signals at the image patch level, enabling the ability for medical diagnosis; Fourth, we conduct thorough experiments to validate the effectiveness of our architecture, and benchmark on numerous public benchmarks e.g., ChestX-ray14, RSNA Pneumonia, SIIM-ACR Pneumothorax, COVIDx CXR-2, COVID Rural, and EdemaSeverity. In both zero-shot and fine-tuning settings, our model has demonstrated strong performance compared with the former methods on disease classification and grounding.
BibTeX
@inproceedings{wu2023medklip,
  author = {Chaoyi Wu and Xiaoman Zhang and Ya Zhang and Yanfeng Wang and Weidi Xie},
  title = {{MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training for X-ray Diagnosis}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01954},
}
Figure