Abstract
Medical vision-language pre-training (VLP) typically relies on matching raw radiology reports with images, a process hindered by linguistic noise and the need for expert domain knowledge. We present MedKLIP, a novel framework that extracts structured clinical triplets—comprising entity, location, and existence—and enriches them with professional medical definitions. This approach allows the model to learn fine-grained visual patterns associated with specific diseases. MedKLIP achieves state-of-the-art performance in zero-shot diagnosis, spatial grounding, and low-data transfer across multiple chest X-ray benchmarks.
Introduction
In general computer vision, models like CLIP benefit from millions of web-scraped image-caption pairs. However, the medical domain faces three significant hurdles:
- Data Scarcity: Paired medical datasets are orders of magnitude smaller than general-purpose datasets.
- Linguistic Complexity: Radiology reports are free-form, containing complex grammar, negations, and stylistic variations that obscure core clinical findings.
- Expertise Gap: Medical concepts are fine-grained; a model must understand the underlying visual attributes of a condition, such as the specific opacities associated with pneumonia.
Existing VLP methods often attempt "coarse-grained" matching of entire reports to images, ignoring the structured domain knowledge available in medical textbooks. MedKLIP addresses these issues by leveraging structured supervision and external medical knowledge.
Methods
MedKLIP introduces three primary innovations to the VLP pipeline:
1. Triplet Extraction
Instead of processing raw text, MedKLIP utilizes tools like RadGraph to convert reports into structured triplets: {entity, position, existence}. For instance, the phrase "Increased right lower lobe opacity" is distilled into {Opacity, Right lower lobe, True}. This removes grammatical noise and focuses the model on clinical facts.
2. Knowledge-enhanced Encoding
To bridge the expertise gap, the model queries medical knowledge bases (e.g., UMLS, Wikipedia) for detailed descriptions of clinical entities. These definitions decompose complex diseases into basic visual attributes, enabling the model to recognize "unseen" diseases by identifying shared attributes—a key driver for zero-shot learning.
3. Transformer-based Fusion
MedKLIP employs a Transformer Decoder where medical entities serve as Queries that attend to image patches (Keys/Values). This architecture naturally aligns specific medical terms with spatial regions in the X-ray, providing inherent explainability through attention maps.
Evaluation and Results
The model was pre-trained on the MIMIC-CXR v2 dataset (227k studies) and evaluated on several public benchmarks, including ChestX-ray14, RSNA Pneumonia, and SIIM-ACR Pneumothorax.
Zero-Shot Performance
MedKLIP significantly outperforms existing state-of-the-art (SOTA) models like CheXzero and BioViL in zero-shot classification. For example, on the SIIM-ACR Pneumothorax dataset, MedKLIP achieved an AUC of 0.8924, compared to CheXzero’s 0.6879.
Grounding and Explainability
The entity-level query mechanism allows MedKLIP to produce accurate heatmaps for disease localization. In the "Pointing Game" metric on the RSNA Pneumonia dataset, MedKLIP scored 0.8721, surpassing GLoRIA's 0.7607.
Robustness and Transfer Learning
MedKLIP demonstrates exceptional data efficiency. When fine-tuning with only 1% of available data, it maintains high performance, indicating that the pre-trained representations are robust and clinically relevant. Furthermore, it shows strong generalization to "unseen" conditions like COVID-19 by leveraging attribute-based definitions.
Conclusion
MedKLIP represents a shift in medical VLP from raw text matching to knowledge-driven, structured supervision. By incorporating medical definitions and clinical triplets, the framework achieves superior generalization and explainability. Future work may address the dependency on initial triplet extraction tools and extend the approach to 3D modalities like CT or MRI.
Summary updated