Introduction
Large Language Models (LLMs) have demonstrated remarkable zero-shot capabilities, yet they often remain "black boxes" accessible only via restricted APIs. This lack of access to internal weights or gradients complicates traditional fine-tuning. While prompting is a common workaround, it is highly sensitive to instruction phrasing and often fails to match the performance of fully supervised models.
To address these challenges, we present CELDA (Clustering-Enhanced Linear Discriminative Analysis), a framework designed to build high-performance text classifiers using black-box LMs without any manually labeled data. CELDA leverages the internal representations of LMs to refine noisy pseudo-labels, training a lightweight and robust classifier that bridges the gap between zero-shot prompting and supervised learning.
Methodology
The CELDA framework operates through a three-stage iterative process:
- Feature Extraction & Pseudo-Labeling: Unlabeled data is passed through a black-box LM to extract hidden state representations ($h$) and initial pseudo-labels ($\hat{y}$) generated via simple prompts.
- Data Cleansing via Clustering and Entropy: Recognizing that pseudo-labels are inherently noisy, CELDA employs K-Means clustering to group semantically similar instances. It then calculates a "Cluster Entropy" score. Clusters with high label variance are deemed unreliable and filtered out, leaving a high-confidence subset of pseudo-labeled data.
- LDA Training and Iteration: A Linear Discriminative Analysis (LDA) model is trained on the cleansed subset. LDA's simplicity provides robustness against remaining noise. The process is iterative: the trained LDA re-labels the full dataset, followed by further cleansing and retraining until convergence.
Experimental Evaluation
CELDA was evaluated across eight benchmark datasets covering sentiment analysis (IMDb, SST-2, Amazon), topic classification (AGNews, DBPedia, Yahoo), and fine-grained intent detection (Banking77, CLINC). These tasks range from binary classification to complex scenarios with up to 150 classes.
Key Results and Insights
CELDA consistently outperforms existing zero-shot and weakly-supervised methods. A primary insight is that while LMs may mislabel individual samples, they excel at grouping semantically related content in their vector space. CELDA exploits this by using clustering to identify and discard areas of model confusion.
As shown in the results, CELDA significantly narrows the performance gap to the "gold standard" supervised models (LDA full). For instance, on the Yahoo dataset using T5-11B, CELDA achieved 71.38% accuracy, a substantial improvement over standard prompting (PET) at 39.66% and Knowledge Prompt-Tuning (KPT) at 63.34%. Furthermore, CELDA scales effectively with the size of the backbone LM, successfully harvesting the superior representations of larger models.
Conclusion and Future Work
CELDA demonstrates that black-box LMs can be transformed into high-accuracy classifiers without human labels by combining semantic clustering with robust statistical modeling. While the method depends on a reasonable initial pseudo-labeling quality, its ability to filter noise makes it a powerful tool for label-scarce environments. Future research will explore sample-wise entropy filtering to further enhance the precision of the data cleansing process.
Summary updated