Prompt-based learning has emerged as a powerful paradigm for adapting pre-trained language models (PLMs) to downstream tasks by framing classification as a "cloze-style" completion task. However, the effectiveness of these models is highly sensitive to two factors: the choice of few-shot instances for fine-tuning and the selection of "verbalizers"—tokens that map model outputs to class labels. In "cold-start" scenarios, where no labeled data is initially available, selecting these components optimally is a significant challenge. Existing methods often treat instance and verbalizer selection as independent problems, which fails to account for their inherent dependency in the model's embedding space.
The COLDSELECT Framework
COLDSELECT is a novel framework that jointly selects instances and verbalizers by modeling data diversity in a shared embedding space. The core intuition is that an instance is more likely to be classified correctly if its representation is proximal to the representation of its corresponding verbalizer token.
The pipeline involves five key stages:
- Shared Embedding Space Construction: Pre-softmax embeddings of the PLM vocabulary and
h[MASK]embeddings of unlabeled instances are mapped into a common space. - Dimensionality Reduction and Clustering: PCA is applied to mitigate the "uniformity" issue in high-dimensional spaces. KMeans clustering is then performed, optimized by a Negative Silhouette Loss to maximize cluster separation and cohesion.
- Cluster Refinement: Outlier clusters (containing only tokens) are discarded, and instances from instance-only clusters are reassigned to ensure every cluster contains both potential verbalizers and data points.
- Selection Metric: A scoring function selects clusters based on Cohesion (density), Separation (distinctness), and Impurity (label diversity/uncertainty).
- Dynamic Updates: As instances are labeled, cluster centroids are updated to the actual embeddings of selected verbalizers, allowing the selection process to adapt to the evolving label distribution.
Experimental Evaluation
The authors evaluated COLDSELECT across eight NLP benchmarks, including sentiment analysis (SST-2, MR, CR, Yelp-full, IMDB), subjectivity classification (Subj), linguistic acceptability (CoLA), and news classification (AG News). These datasets represent a wide range of domains and class distributions.
Results and Key Insights
COLDSELECT consistently outperforms state-of-the-art baselines, such as PATRON. Notably, on the IMDB dataset with a 32-instance budget, COLDSELECT achieved 87.37% accuracy, representing a 1.9% improvement over the next best method. The framework demonstrates significant robustness to class imbalance, particularly on the Yelp-full dataset.
A key insight from this work is the Dependency Principle: instance-label probabilities are tied to verbalizer token proximity. By selecting instances and verbalizers that are "near" each other in the PLM's internal space, the model reduces noise and improves learning efficiency. Furthermore, COLDSELECT often achieves higher accuracy with smaller budgets (e.g., 32 or 64 instances) than baseline methods do with 128, underscoring that the quality of selected data is more critical than quantity.
Conclusion and Future Work
COLDSELECT provides a robust solution for cold-start prompt-based learning by unifying instance and verbalizer selection. While currently reliant on linear transformations like PCA and standard clustering techniques, future work could explore non-linear dimensionality reduction and more complex clustering algorithms. Additionally, extending the framework to multilingual and multimodal settings remains a promising direction for future research.
Summary updated