Multiple Prompt Fusion for Zero-Shot Lesion Detection Using Vision-Language Models

Miaotian Guo, Huahui Yi, Ziyuan Qin, Haiying Wang, Aidong Men, Qicheng Lao

Abstract

The success of large-scale pre-trained vision-language models (VLM) has provided a promising direction of transferring natural image representations to the medical domain by providing a well-designed prompt with medical expert-level knowledge. However, one prompt has difficulty in describing the medical lesions thoroughly enough and containing all the attributes. Besides, the models pre-trained with natural images fail to detect lesions precisely. To solve this problem, fusing multiple prompts is vital to assist the VLM in learning a more comprehensive alignment between textual and visual modalities. In this paper, we propose an ensemble guided fusion approach to leverage multiple statements when tackling the phrase grounding task for zero-shot lesion detection. Extensive experiments are conducted on three public medical image datasets across different modalities and the detection accuracy improvement demonstrates the superiority of our method.
BibTeX
@inproceedings{guo2023multiple,
  author = {Miaotian Guo and Huahui Yi and Ziyuan Qin and Haiying Wang and Aidong Men and Qicheng Lao},
  title = {{Multiple Prompt Fusion for Zero-Shot Lesion Detection Using Vision-Language Models}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43904-9\_28},
}

Loading related content...

Figure