PCLmed: Champion Solution for ImageCLEFmedical 2024 Caption Prediction Challenge via Medical Vision-Language Foundation Models

Bang Yang, Yue Yu, Yuexian Zou, Tong ZhangCLEF (Working Notes) 2024 · OpenReview

PCLmed: Champion Solution for ImageCLEFmedical 2024 Caption Prediction Challenge

Automated medical report generation (MRG) is a critical frontier in healthcare AI, aimed at alleviating the heavy workload of radiologists. However, the domain faces significant hurdles: the scarcity of high-quality paired medical data and the limitations of current Vision-Language Foundation Models (VLFMs) in capturing specialized clinical nuances. PCLmed, the winning solution for the ImageCLEFmedical 2024 Caption Prediction Challenge, addresses these challenges through a sophisticated ensembling strategy and a novel modality-aware adaptation technique.

The PCLmed Architecture

The core of PCLmed is a BLIP-2 style framework enhanced with two primary innovations: Vision Encoder Ensembling (VEE) and Modality-Aware Adaptation (MAA).

The complete PCLmed architecture, featuring Vision Encoder Ensembling and Modality-Aware Adaptation within the LLM.

Vision Encoder Ensembling (VEE)

Rather than relying on a single vision backbone, PCLmed leverages the complementary strengths of two distinct models:

  1. EVA ViT-g: A massive general-purpose vision model that captures broad structural and global information.
  2. BiomedCLIP: A model specifically pre-trained on 15 million medical image-text pairs, providing deep clinical domain knowledge.

By concatenating features from both encoders, the system provides the Large Language Model (LLM) with a more holistic visual representation of the medical image.

Modality-Aware Adaptation (MAA)

A common issue in multi-modal fine-tuning is "modality interference," where vision and text features compete for the same parameter space in standard adapters. PCLmed introduces V-Adapters and T-Adapters—separate parameter-efficient fine-tuning (PEFT) modules placed after each block of the LLM (Pangu-$\alpha$). This allows the model to calibrate visual and linguistic signals independently, facilitating better cross-modal collaboration.

Data and Training Strategy

The model was developed using the ROCOv2 (Radiology Objects in COntext version 2) dataset, which contains over 70,000 training images across various modalities like CT and X-ray.

Word cloud visualizing the frequent medical terms and concepts present in the ROCOv2 dataset.

To ensure a stable training process, the researchers employed a zero-initialization strategy for the new adapters. This prevents the initial random weights of the adapters from disrupting the LLM's pre-trained knowledge, allowing for a smooth transition to the specialized medical captioning task.

Performance and Insights

PCLmed secured first place in the ImageCLEFmedical 2024 challenge, achieving a BERTScore of 0.6299 on the test set. Beyond quantitative metrics, the model showed a marked improvement in qualitative accuracy. It successfully distinguished between imaging modalities (e.g., CT vs. X-ray) and accurately identified complex clinical findings such as "bilateral hilar lymphadenopathy" or "internal jugular vein thrombosis," which often baffle standard models.

While highly effective, the authors identify future work in resolving spatial confusion (e.g., left vs. right) and increasing input resolution to capture even finer clinical details. PCLmed stands as a robust blueprint for the next generation of medical vision-language systems.

Summary updated

BibTeX
@inproceedings{yang2024pclmed,
  author = {Bang Yang and Yue Yu and Yuexian Zou and Tong Zhang},
  title = {{PCLmed: Champion Solution for ImageCLEFmedical 2024 Caption Prediction Challenge via Medical Vision-Language Foundation Models}},
  booktitle = {CLEF (Working Notes)},
  year = {2024},
  url = {https://openreview.net/forum?id=0JohJIwXTq},
}
Figure