Grounded knowledge-enhanced medical vision-language pre-training for chest X-ray

Qiao Deng, Zhongzhen Huang, Yunqi Wang, Zhichuan Wang, Zhao Wang, Xiaofan Zhang, Qi Dou, Yeung Yu Hui, Edward S. Hui

TL;DR

This paper introduces GK-MVLP, a novel framework for pre-training medical vision-language models on chest X-rays. By grounding medical knowledge to specific anatomical regions within the X-ray and leveraging this "grounded knowledge" during pre-training, GK-MVLP significantly improves performance on various downstream tasks like disease classification, localization, report generation, and visual question answering, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Medical foundation models hold immense promise for revolutionizing healthcare by providing robust representations of medical data. Vision-language pre-training (VLP) is a key approach for this, aiming to align medical images with their corresponding textual descriptions. However, existing VLP methods often struggle with the inherent complexity and variability of medical data. Specifically, they can be marred by redundant information, leading to suboptimal alignment between images and reports. This is particularly challenging in chest X-rays, where subtle abnormalities and nuanced terminology require deep contextual understanding. Previous attempts to incorporate external medical knowledge have sometimes introduced irrelevant information, confusing the model. This paper addresses the need for a more precise and effective way to align medical images and text, specifically by leveraging structured medical knowledge in a grounded manner.

Methods

The core innovation of GK-MVLP lies in its "grounded knowledge-enhanced module." This module aims to precisely align medical knowledge with relevant anatomical regions in chest X-rays. The process involves:

  • Medical Knowledge Prompts: Medical knowledge is first converted into structured "prompts" that include the entity (e.g., pneumonia), its anatomical position (e.g., right lower lung zone), and its existence. This is derived from radiology reports and annotated labels.
  • Anatomical Region Detection: A pre-trained detector identifies anatomical regions within the chest X-ray.
  • Grounded Knowledge-Enhanced Module: This module uses a transformer-based cross-attention mechanism to fuse region-based visual features, medical knowledge prompts, and global image features. This allows for fine-grained alignment, ensuring that medical knowledge is associated with the correct visual context.
  • Entity Classification Loss: An additional loss function encourages the model to correctly identify the presence or absence of specific clinical entities within the image, further refining the alignment.

The overall GK-MVLP framework builds upon the BLIP architecture, incorporating this new module for pre-training.

Datasets

  • Pre-training: MIMIC-CXR dataset, containing paired chest X-ray images and radiology reports. Medical knowledge prompts were generated using anatomical annotations from the Chest ImaGenome dataset.
  • Downstream Tasks:
    • Disease Classification: NIH Chest X-ray dataset, RSNA Pneumonia dataset, and CheXpert dataset.
  • Disease Localization: RSNA Pneumonia dataset.
  • Report Generation: IU X-Ray dataset.
  • Medical Visual Question Answering (VQA): VQA-RAD dataset and SLAKE dataset.

Key Insights and Ideas

  • Grounding is Key: The paper strongly suggests that simply aligning global image and text features, or even local patches, is insufficient for complex medical tasks. Grounding medical knowledge to specific anatomical regions is crucial for precise understanding.
  • Knowledge Prompts Enhance Specificity: By structuring medical knowledge into prompts with positional information, the model can avoid irrelevant information and focus on clinically relevant details.
  • Fusion of Global and Local Context: The GK module effectively combines global image features with localized anatomical and knowledge-based information, leading to a more comprehensive representation.
  • Improved Performance Across Diverse Tasks: The framework's success across classification, localization, generation, and VQA highlights its robustness and generalizability as a foundation model.

Key Results

GK-MVLP consistently achieved state-of-the-art (SOTA) or competitive performance across various downstream tasks, often with significantly smaller pre-training datasets compared to other SOTA models.

  • Disease Classification: Outperformed CNN-based SOTA models and ranked first or second among ViT-based models on RSNA Pneumonia and CheXpert datasets. For example, on the RSNA dataset with 100% training data, GK-MVLP achieved an AUROC of 92.8%, surpassing many others.
  • Disease Localization: Achieved SOTA performance on the RSNA Pneumonia dataset, with a mean average precision (mAP) of 31.2% using 100% of the training data, outperforming methods like PRIOR and MGCA.
  • Report Generation: Outperformed SOTA models on most metrics (BLEU4, METEOR, CIDEr) on the IU X-Ray dataset, achieving a CIDEr score of 0.713.
  • Medical VQA: Demonstrated superior performance on both VQA-RAD and SLAKE datasets, achieving the highest overall accuracies of 72.5% and 82.5%, respectively.

Limitations and Future Work

  • Limited Dataset Size: The experiments were conducted using a relatively limited amount (160k) of well-annotated image-report pairs.
  • Modality Extension: The authors suggest extending the model to other imaging modalities like CT, MRI, and ultrasound to further improve its generalizability.
  • Large Language Model Integration: Future work could explore how large language models can provide more useful information for processing medical images from different modalities.

Summary updated

Abstract

Medical vision-language pre-training has emerged as a promising approach for learning domain-general representations of medical image and text. Current algorithms that exploit the global and local alignment between medical image and text could however be marred by the redundant information in medical data. To address this issue, we propose a grounded knowledge-enhanced medical vision-language pre-training (GK-MVLP) framework for chest X-ray. In this framework, medical knowledge is grounded to the appropriate anatomical regions by using a transformer-based grounded knowledge-enhanced module for fine-grained alignment between anatomical region-level visual features and the textural features of medical knowledge. The performance of GK-MVLP is competitive with or exceeds the state of the art on downstream chest X-ray disease classification, disease localization, report generation, and medical visual question-answering tasks. Our results show the advantage of incorporating grounding mechanism to remove biases and improve the alignment between chest X-ray image and radiology report.
BibTeX
@article{deng2026grounded,
  author = {Qiao Deng and Zhongzhen Huang and Yunqi Wang and Zhichuan Wang and Zhao Wang and Xiaofan Zhang and Qi Dou and Yeung Yu Hui and Edward S. Hui},
  title = {{Grounded knowledge-enhanced medical vision-language pre-training for chest X-ray}},
  journal = {Biomedical Signal Processing and Control},
  year = {2026},
  doi = {10.1016/j.bspc.2025.108416},
}
Figure