Multiple Prompt Fusion for Zero-Shot Lesion Detection Using Vision-Language Models
Lists
Only you see this
Abstract
The success of large-scale pre-trained vision-language models (VLM) has provided a promising direction of transferring natural image representations to the medical domain by providing a well-designed prompt with medical expert-level knowledge. However, one prompt has difficulty in describing the medical lesions thoroughly enough and containing all the attributes. Besides, the models pre-trained with natural images fail to detect lesions precisely. To solve this problem, fusing multiple prompts is vital to assist the VLM in learning a more comprehensive alignment between textual and visual modalities. In this paper, we propose an ensemble guided fusion approach to leverage multiple statements when tackling the phrase grounding task for zero-shot lesion detection. Extensive experiments are conducted on three public medical image datasets across different modalities and the detection accuracy improvement demonstrates the superiority of our method.
BibTeX
@inproceedings{guo2023multiple,
author = {Miaotian Guo and Huahui Yi and Ziyuan Qin and Haiying Wang and Aidong Men and Qicheng Lao},
title = {{Multiple Prompt Fusion for Zero-Shot Lesion Detection Using Vision-Language Models}},
booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
year = {2023},
doi = {10.1007/978-3-031-43904-9\_28},
}Loading related content...