UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models

Xin Li, Sima Behpour, Thang Doan, Wenbin He, Liang Gou, Liu Ren

Abstract

In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.

Introduction

In the era of "data-hungry" deep learning, data labeling remains the ultimate bottleneck. Traditionally, researchers have relied on Active Learning (AL) or Semi-Supervised Learning (SSL) to mitigate labeling costs. However, these methods typically require a "seed" set of labeled data or a predefined downstream task to guide the selection process.

This paper addresses a more challenging "cold-start" scenario: Data Pre-Selection. In this setting, an practitioner must select a representative subset of images from a massive unlabeled pool for labeling, without knowing the specific downstream task (e.g., whether the goal is fine-grained classification or general object detection). While previous methods relied on visual features from models like CLIP, they often struggled to separate complex categories without textual guidance. UP-DP (Unsupervised Prompt learning for Data Pre-selection) bridges this gap by leveraging the joint feature space of Vision-Language Models (VLMs).

The UP-DP Framework

UP-DP introduces a novel way to utilize VLMs for data selection without requiring human-written prompts or labeled data. The framework is built on three core pillars:

1. Frozen Foundation with Learnable Prompts

The researchers utilize BLIP-2 as the backbone model. To preserve the extensive pre-trained knowledge of the VLM while adapting it to the data selection task, the model parameters remain frozen. Instead of static, human-defined prompts (e.g., "a photo of a [class]"), UP-DP introduces a set of learnable context vectors—continuous embeddings that are optimized through an unsupervised process.

2. Dual Contrastive Learning

To train these prompts without ground-truth labels, UP-DP employs two distinct loss functions:

  • Instance-level Contrastive Loss: This ensures that different augmentations (views) of the same image remain close in the feature space, promoting robust feature extraction.
  • Cluster-level Contrastive Loss: This forces the model to organize the entire dataset into a diverse cluster structure. By preventing the feature space from collapsing into a single "cloud," the model creates a structured map of the data distribution.

3. Medoid Selection

Once the prompts are learned and the feature space is structured, the model performs clustering. It then selects the "medoid"—the most central and representative point—of each cluster for labeling. This ensures that the selected subset covers the maximum diversity of the original dataset.

Key Insights and Results

The "Scattering" Effect

A critical finding of this research is the "scattering" effect. Unimodal (vision-only) features often become mixed when class information is unknown. By introducing learned multimodal prompts, the features become more distinct and "scattered," making it significantly easier to identify representative samples across different classes.

Performance Gains

UP-DP was evaluated across seven diverse benchmark datasets, including Caltech101 (generic objects), OxfordPets (fine-grained), and EuroSAT (satellite imagery). Key results include:

  • Superior Accuracy: UP-DP consistently outperformed the state-of-the-art (USL) by an average of 7%, with gains reaching 20% in specific fine-grained settings.
  • Class Balance: Visualizations on the EuroSAT dataset demonstrated that UP-DP's selection is much more balanced across classes compared to random sampling, which frequently misses entire categories.
  • Robustness: The performance improvements were consistent across various CLIP backbones, from ResNet-50 to ViT-G.

Plug-and-Play Generalizability

One of the most surprising outcomes was the generalizability of the learned prompts. Prompts trained on one dataset (e.g., OxfordPets) could be directly applied to a completely different domain (e.g., Flowers102) to improve feature extraction. This suggests that the prompts learn fundamental ways to "interrogate" visual data rather than simply over-fitting to a specific domain.

Limitations and Future Work

Despite its success, UP-DP faces challenges regarding interpretability. Because the learned prompts are continuous vectors in a high-dimensional space, they function as "black boxes." While mapping these vectors to the nearest human words yielded terms like "wood" or "plane," the full logic of the learned prompt remains difficult to decode. Additionally, while UP-DP reduces bias, extremely small annotation budgets still pose a risk of selection bias.

Future research will focus on making these prompts more transparent and extending the framework to more complex data modalities, such as video and 3D point clouds.

Summary updated

Abstract

In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks with a limited annotation budget. Previous approaches to data pre-selection relied solely on visual features extracted from foundation models, such as CLIP and BLIP-2, but largely ignored the powerfulness of text features. In this work, we argue that, with proper design, the joint feature space of both vision and text can yield a better representation for data pre-selection. To this end, we introduce UP-DP, a simple yet effective unsupervised prompt learning approach that adapts vision-language models, like BLIP-2, for data pre-selection. Specifically, with the BLIP-2 parameters frozen, we train text prompts to extract the joint features with improved representation, ensuring a diverse cluster structure that covers the entire dataset. We extensively compare our method with the state-of-the-art using seven benchmark datasets in different settings, achieving up to a performance gain of 20%. Interestingly, the prompts learned from one dataset demonstrate significant generalizability and can be applied directly to enhance the feature extraction of BLIP-2 from other datasets. To the best of our knowledge, UP-DP is the first work to incorporate unsupervised prompt learning in a vision-language model for data pre-selection.
BibTeX
@inproceedings{li2023up,
  author = {Xin Li and Sima Behpour and Thang Doan and Wenbin He and Liang Gou and Liu Ren},
  title = {{UP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=VMAgvbBBts},
}
Figure