Abstract
Vision-Language Pre-training (VLP) has significantly advanced cross-modal tasks, yet general-purpose models often struggle with the specialized requirements of the medical domain. We present Clinical-BERT, a specialized VLP framework designed specifically for radiology. By integrating domain-specific pre-training tasks—Clinical Diagnosis (CD), Masked MeSH Modeling (MMM), and Image-MeSH Matching (IMM)—Clinical-BERT learns to prioritize clinically significant terms and regions. Our results demonstrate that Clinical-BERT achieves state-of-the-art performance in both automated radiograph diagnosis and the generation of high-quality clinical reports.
Introduction
While VLP models trained on natural images and language have excelled in general tasks like image captioning, they face a significant "domain gap" when applied to radiology. In medical imaging, terminology is precise and non-interchangeable; for instance, specific Medical Subject Headings (MeSH) like "Pleural Effusion" carry critical diagnostic weight that general synonyms cannot replace. Furthermore, radiologists interpret images by scanning specific regions for evidence of pathology, a process that standard VLP models—which treat all image regions and words with equal importance—fail to replicate. Clinical-BERT addresses these challenges by encoding medical expertise directly into the pre-training phase.
Methodology
Clinical-BERT utilizes a single-stream transformer architecture based on BERT-base, integrating visual features extracted from a DenseNet121 backbone with textual report data. The model is optimized through three novel pre-training tasks designed to capture the semantic structure of medical knowledge.
Domain-Specific Pre-training Tasks
- Clinical Diagnosis (CD): The model is trained to predict 14 disease categories from radiographs, establishing a foundation of medical "common sense."
- Masked MeSH Modeling (MMM): Unlike standard random masking, MMM specifically targets MeSH keywords. This forces the model to prioritize learning the most clinically significant terms within a report.
- Image-MeSH Matching (IMM): This task aligns image regions with medical keywords using a specialized two-level sparse attention mechanism.
Two-Level Sparse Attention
To mimic a radiologist's focused expertise, Clinical-BERT employs two levels of sparsity:
- Region Sparse Attention (RSA): This mechanism selects only the top-$K$ most relevant image regions for a given word, effectively filtering out visual noise.
- Word Sparse Attention (WSA): This boosts the contribution of MeSH words when calculating the alignment score between an image and its corresponding report.
Experiments and Results
Datasets
The model was pre-trained on the MIMIC-CXR dataset (377,000 images, 227,000 reports). Evaluation for report generation was conducted on the IU X-Ray and COV-CTR datasets, while diagnostic performance was tested on the NIH ChestX-ray14 dataset.
Key Findings
Clinical-BERT achieved state-of-the-art (SOTA) results across multiple benchmarks:
- Report Generation: On the IU X-Ray dataset, the model outperformed previous benchmarks across all standard NLG metrics (BLEU, METEOR, ROUGE-L, CIDEr).
- Clinical Efficacy: On MIMIC-CXR, Clinical-BERT showed an 8.6% improvement in diagnostic accuracy for generated reports compared to the previous leading model, R2Gen.
- Diagnostic Accuracy: On the NIH dataset, the model achieved the best results for 9 out of 14 pathologies, reaching a mean AUC of 0.8450. This confirms that vision-language pre-training enhances the model's ability to detect diseases visually.
Discussion and Conclusion
The success of Clinical-BERT highlights that domain-specific pre-training is essential for clinical accuracy. By treating MeSH words as semantic anchors and utilizing sparse attention to mimic expert focus, the model bridges the gap between general AI and specialized medical needs.
Future work will explore incorporating explicit organ localization to further refine MeSH word prediction and extending this framework to broader tasks such as medical Visual Question Answering (VQA) and image-report retrieval. Clinical-BERT represents a significant step toward AI systems that can truly understand the language of radiology.
Summary updated