UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models

Xin Li, Sima Behpour, Thang Doan, Wenbin He, Liang Gou, Liu Ren

Introduction: The Data Pre-Selection Challenge

Deep learning is notoriously data-hungry, but human annotation is expensive. Traditional data-efficient paradigms like Active Learning (AL) or Semi-Supervised Learning (SSL) help, but they come with a catch: they require an initial set of labeled data and assume you already know what your downstream task is (e.g., classifying specific dog breeds).

But what if you have a massive pool of unlabeled data, a tiny labeling budget, and you don't yet know what the downstream tasks will be? This is the challenge of data pre-selection. The goal is to make a single pass through an unlabeled dataset and select a small, highly diverse, and representative subset to be annotated, maximizing performance for any future, undefined task.

Comparison of data pre-selection with traditional learning paradigms, alongside UMAP visualizations showing how learned prompts improve feature clustering.

Previous approaches tackled this by clustering the visual features of foundation models (like CLIP). However, they completely ignored the text modality, leaving the rich, joint vision-language representation space of these models untapped. To address this, researchers introduce UP-DP (Unsupervised Prompt Learning for Data Pre-selection), a novel method that identifies the best unlabeled examples to annotate by extracting rich multimodal features to cluster and select highly diverse data points.

Methodology: Unsupervised Prompt Learning

Instead of relying solely on image features, UP-DP leverages the joint vision-text space of the BLIP-2 foundation model.

The UP-DP framework illustrating the data pre-selection workflow, unsupervised prompt learning via contrastive losses, and prompt generalization across datasets.

The UP-DP framework operates through several key steps:

  1. Frozen Base Model: The heavy BLIP-2 model (specifically its image encoder and Q-Former) is kept completely frozen to preserve its foundational knowledge.
  2. Learnable Prompts: The authors introduce continuous, learnable context vectors (prompts) that are fed into the text side of BLIP-2 alongside the image.
  3. Unsupervised Contrastive Clustering: Because there are no labels, the prompts cannot be trained using standard cross-entropy. Instead, the prompts are trained alongside two lightweight MLP heads using an unsupervised contrastive learning objective. An instance-level head maximizes similarity between augmented views of the same image, while a cluster-level head groups similar images into predefined clusters.
  4. Selection: Once the prompts are trained, every image in the dataset is assigned to a cluster. UP-DP then selects the "medoid" (the most central, representative instance) from each cluster to be sent to human annotators.

Key Insights and Generalization

The authors evaluated UP-DP across a diverse benchmark of seven image classification datasets, including generic objects (Caltech101), human actions (UCF101), fine-grained categories (OxfordPets, Flowers102, FGVCAircraft), textures (DTD), and satellite imagery (EuroSAT).

Several fascinating insights emerged from this evaluation:

  • Multimodal features outperform unimodal features: Even without text labels, feeding a learned text prompt into a Vision-Language model forces it to extract a joint multimodal feature. This joint feature separates underlying classes much better than vision-only features.
  • Prompts can be learned without labels: Supervised data is not strictly necessary for prompt tuning. Unsupervised contrastive clustering provides a strong enough signal to learn prompts that extract highly discriminative features.
  • Mind-blowing Cross-Domain Generalization: The most surprising finding is that prompts learned on one dataset work incredibly well on entirely different datasets. For example, a prompt learned purely on dog and cat images (OxfordPets) can be plugged into BLIP-2 to extract superior features for flower classification (Flowers102). The learned prompts seem to act as universal "feature extraction instructions" rather than memorizing dataset-specific semantics.
  • Shorter prompts prevent overfitting: A short context length (just 4 learnable tokens) generalized better to unknown downstream tasks than longer contexts (16 tokens).

Key Results

  • Quantitative Performance: When the pre-selected data was labeled and used to train a downstream linear probe (using CLIP), UP-DP consistently outperformed random selection and state-of-the-art unsupervised selective labeling (USL) baselines. It achieved an average accuracy boost of ~7% across the 7 datasets, with gains up to 20% on specific datasets.
  • Semantic Segmentation: UP-DP isn't just for classification. When used to select 100 images from Pascal VOC to train a DINOv2 segmentation model, UP-DP achieved a 57.0 mIoU, beating random selection (53.5 mIoU).
  • Qualitative Clustering: UMAP visualizations showed that UP-DP successfully clustered complex, unlabeled variations. For example, it naturally separated images of Shiba Inus into distinct clusters based on background (grass vs. concrete) and posture (standing vs. lying down), ensuring the final selected subset was highly diverse.

Limitations and Future Work

The authors candidly acknowledge a few limitations. First, because the learned prompts are continuous vectors, they act as a "black box." When mapped back to the nearest English words, the results were mostly gibberish (e.g., "makes wood takes healthy"), making it hard to interpret why these specific vectors trigger such great feature extraction. Second, UP-DP operates under extreme budget constraints (e.g., selecting only 200 images from a massive dataset). Even with smart clustering, selecting just one instance per cluster risks omitting rare edge cases, potentially leading to biased downstream models.

Future work will likely explore how to make these continuous prompts more interpretable and how to further mitigate selection bias in ultra-low-budget scenarios.

Summary updated

Abstract

In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20\%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.
BibTeX
@inproceedings{li2023up,
  author = {Xin Li and Sima Behpour and Thang Doan and Wenbin He and Liang Gou and Liu Ren},
  title = {{UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=VMAgvbBBts},
}
Figure