Abstract
In the field of medical decision-making, precise anomaly detection in medical imaging plays a pivotal role in aiding clinicians. However, previous work is reliant on large-scale datasets for training anomaly detection models, which increases the development cost. This paper focuses on the task of medical image anomaly detection in the few-shot setting, which is critically significant for the medical field where data collection and annotation are both very expensive. We propose an innovative approach, MediCLIP, which adapts the CLIP model to few-shot medical image anomaly detection through self-supervised fine-tuning. By designing a series of medical image anomaly synthesis tasks to simulate common disease patterns, we transfer the powerful generalization capabilities of CLIP to medical imaging. When only few-shot normal medical images are provided, MediCLIP achieves state-of-the-art performance in anomaly detection and localization compared to other methods across three distinct medical imaging modalities.
Introduction
In clinical practice, "anomaly detection" (AD) is the fundamental task of identifying deviations from healthy physiological states in medical scans and precisely localizing the suspicious regions. While deep learning has revolutionized this field, most high-performing models require massive, meticulously labeled datasets. In the medical domain, however, data is expensive to acquire, and scans showing rare diseases or specific lesions are often scarce.
This research addresses the challenge of few-shot medical image anomaly detection, where a model must learn to identify pathologies after seeing only a handful (e.g., 4 to 32) of healthy images. While general-purpose vision-language models like CLIP (Contrastive Language-Image Pre-training) excel at zero-shot tasks in natural imagery, they often struggle with the subtle, low-contrast textures and specific structural nuances of medical imaging. MediCLIP bridges this gap by adapting CLIP's broad knowledge to the specialized requirements of clinical diagnostics.
Methodology
MediCLIP transforms the standard CLIP architecture into a specialized medical anomaly detector through three core innovations: learnable prompting, multi-scale adapters, and synthetic anomaly generation.
1. Learnable Prompting
Instead of relying on static, hand-crafted text prompts like "a photo of a healthy lung," MediCLIP utilizes learnable tokens. These are mathematical representations that the model optimizes during training to capture the nuanced definitions of "healthy/normal" versus "disease/abnormal" within a medical context. This allows the text encoder to provide more precise semantic guidance for the vision encoder.
2. Multi-Scale Adapters
Standard CLIP outputs are often too coarse for the fine-grained localization required in medicine. To solve this, the researchers integrated "adapters"—small, trainable layers—into the intermediate levels of CLIP’s vision encoder. By extracting features from multiple stages of the network, the model can simultaneously analyze broad anatomical structures and minute textural details, such as tiny fractures or subtle intensity shifts.
3. Multi-Task Anomaly Synthesis
Because the model is trained primarily on healthy images, it must "imagine" what an anomaly looks like to develop its detection capabilities. MediCLIP employs three specific synthesis tasks to simulate real-world pathology:
- CutPaste: Pasting image patches to simulate foreign objects or structural breaks.
- GaussIntensityChange: Altering local brightness and contrast to mimic the appearance of tumors or cysts.
- Source: Deforming anatomical shapes to simulate organ enlargement, such as cardiomegaly in chest X-rays.
Experimental Setup and Datasets
The model was rigorously evaluated across three distinct medical imaging modalities to demonstrate its versatility and robustness:
- CheXpert: Chest X-rays used for detecting 12 different pulmonary and cardiac diseases.
- BrainMRI: 2D brain scans focused on tumor detection.
- BUSI: Breast ultrasound images categorized into normal, benign, and malignant classes.
Key Results and Insights
MediCLIP demonstrates significant improvements over existing state-of-the-art (SOTA) methods like PatchCore and RegAD. In few-shot settings, MediCLIP outperformed these methods by approximately 10% in Image-AUROC, a standard metric for detection accuracy.
One of the most striking findings is the model's data efficiency. On the CheXpert dataset, MediCLIP achieved 94% of the performance of "full-shot" models (trained on thousands of images) while using less than 1% of the training data. Furthermore, the model exhibits strong cross-task generalization; a version trained on chest X-rays can effectively identify anomalies in brain MRIs without additional fine-tuning, suggesting it has learned a universal concept of "medical abnormality."
Beyond simple classification, MediCLIP provides actionable insights for clinicians by generating highly accurate anomaly maps. These maps highlight the specific areas of concern, facilitating faster and more reliable diagnostic reviews.
Conclusion and Future Work
MediCLIP represents a major step forward in making medical AI more accessible and cost-effective by reducing the reliance on large, annotated datasets. By combining the foundational power of CLIP with specialized adapters and clever anomaly synthesis, it sets a new benchmark for few-shot medical anomaly detection.
Future research will focus on increasing the complexity of synthetic anomalies using generative models (like GANs or Diffusion models) to better mimic biological textures. Additionally, extending the framework to handle 3D volumetric data, such as CT scans and full MRI volumes, remains a critical next step for clinical deployment.
Summary updated