On Unsupervised Prompt Learning for Classification with Black-box Language Models

Zhen-Yu Zhang, Jiandong Zhang, Huaxiu Yao, Gang Niu, Masashi SugiyamaSubmission to ICLR 2025 · Withdrawn

TL;DR

Fine-tuning commercial, black-box LLMs usually requires manually labeled data—an expensive bottleneck. This paper introduces PPD (Prompt learning with Pseudo-labeled Demonstrations), an unsupervised prompt optimization framework that simultaneously learns discrete prompt tokens and discovers high-quality pseudo-labels for unlabeled datasets. By incorporating retrieved in-context demonstrations directly into the black-box training objective, PPD consistently outperforms zero-shot inference and existing prompt-tuning baselines on GLUE and MMLU benchmarks.

What Problem Does This Paper Address?

Top-tier large language models (LLMs) such as GPT-4 are predominantly deployed as black boxes: users interact with them via commercial APIs without access to model weights, latent activations, or backpropagation gradients. Adapting these models to downstream classification tasks typically demands labeled datasets for fine-tuning or prompt optimization.

While LLMs are surprisingly capable zero-shot annotators, using naive self-training (labeling data and immediately fine-tuning) risks severe overfitting on noisy labels. Furthermore, existing black-box prompt-tuning algorithms optimize prompts in isolation and only prepend few-shot demonstrations at inference time. This creates a fundamental disconnect between how prompts are trained and how they are actually deployed with in-context learning (ICL).

Methods

The authors introduce PPD, which optimizes a discrete prompt and pseudo-labels concurrently without requiring labeled data:

[Unlabeled Data] 
       │
       ▼
1. Confident Pseudo-Label Identification (Linguistic or Bias-Reduced Confidence)
       │
       ▼
2. Dynamic In-Context Demonstration Construction (K-NN retrieval + prompt-assigned labels)
       │
       ▼
3. Joint Prompt Optimization via Policy Gradients (VR-PGE) + Entropy Regularization
  1. Reliable Pseudo-Label Identification: Using zero-shot API outputs, the method calculates prediction confidence scores either via multi-temperature average linguistic confidence ($c^{LG}$) or bias-reduced confidence ($c^{RD}$, which normalizes against random-context priors). Instances exceeding a threshold ($\gamma = 0.7$) are designated as initial anchors.
  2. Dynamic In-Context Demonstrations: For any query sample, $K$-nearest neighbors are retrieved using sentence embeddings. The black-box model predicts their pseudo-labels on the fly under the current prompt, and these pairs serve as in-context demonstrations ($D$).
  3. Training Objective: The discrete prompt $z$ is optimized to ensure that when the model evaluates a reliable sample conditioned on the learned prompt and the pseudo-labeled demonstrations, its prediction matches the sample's reliable pseudo-label: $$\mathcal{L}_{\text{main}}(z) = \sum_{l=1}^n \mathbf{1}[c_l \ge \gamma] \cdot \ell\big(f(x_l, z, D_l),\, f(x_l, \emptyset, \emptyset)\big)$$
  4. Entropy Regularization & Optimization: Each token position in prompt $z$ is parameterized as a categorical distribution over a candidate vocabulary mined via Pointwise Mutual Information (PMI). To avoid flat, ambiguous token distributions at test time, an entropy minimization penalty ($\alpha \sum H(p_i)$) forces distributions to become sharp and deterministic. The discrete parameters are updated via a Variance-Reduced Policy Gradient Estimator (VR-PGE) and projected stochastic gradient descent.

Datasets

The authors evaluate PPD across two major benchmark suites:

  • GLUE: CoLA, SST-2, QQP, MRPC, MNLI, WNLI, and RTE.
  • MMLU (9 subsets): Astronomy (AST), High-School Computer Science (HSCS), High-School Mathematics (HSM), College Mathematics (CMath), College Computer Science (CCS), College Medicine (CMed), Management (MAN), Marketing (MAR), and All-Random (RND).

No new datasets are introduced; all evaluation uses standardized splits without ground-truth labels during prompt learning.

Key Insights and Ideas

  • Bridging the Train-Test Gap in ICL: Standard prompt learning optimizes a prompt alone and later appends demonstrations at test time. PPD places pseudo-labeled demonstrations inside the prompt-learning loop, ensuring the prompt is optimized specifically to exploit demonstration context.
  • Mutual Reinforcement: Prompt learning and pseudo-labeling form a virtuous cycle: better prompts yield more accurate in-context demonstration labels, which in turn steer the policy gradient updates toward better prompts.
  • Entropy Penalization Prevents Prompt Degeneracy: Without minimizing Shannon entropy across token categorical distributions, learned distributions remain dispersed, leading to erratic prompt sampling during inference.

Key Results

Experiments conducted using OpenAI’s gpt-4 and gpt-4o-mini show that PPD consistently outperforms zero-shot prediction (Direct), standard pseudo-labeled few-shot inference (ICL), and adapted black-box prompt baselines (BDPL, RLPrompt):

  • GLUE: On SST-2, PPD (RD) achieves 91.5% accuracy versus 89.5% for Direct and 90.2% for BDPL (RD). On MNLI, PPD reaches 92.1% (vs. 90.2% for ICL-RD).
  • MMLU: On challenging reasoning tasks, PPD demonstrates substantial gains over Direct prompting:
  • College Mathematics (CMath): 44.9% vs. 40.5% (Direct).
  • College Computer Science (CCS): 72.8% vs. 68.2% (Direct).
  • Management (MAN): 79.8% vs. 76.6% (Direct).
  • Demonstration Scaling: Testing $K \in \{3, 5\}$ revealed that $K=5$ paired with cross-entropy loss yielded the most consistent improvements across tasks.
  • Model Portability: Both gpt-4 and the lighter gpt-4o-mini exhibited consistent accuracy gains when evaluated with PPD prompts over direct zero-shot prompting.

Limitations and Future Work

  • API Query Costs: Estimating gradients via VR-PGE requires repeated sampling across batch iterations and multiple forward passes per prompt token, which can become expensive on proprietary APIs.
  • Sensitivity to Zero-Shot Baselines: The framework relies on the underlying LLM having sufficient baseline capability to yield well-calibrated confidence scores. If initial predictions on an obscure task are poor, noise may propagate into the demonstration pool.
  • Downstream Fine-Tuning Integration: While the authors position the generated pseudo-labeled dataset as an asset for subsequent API fine-tuning, the paper primarily evaluates inference-time prompt accuracy rather than post-fine-tuning downstream checkpoints.

Summary updated

Abstract

Large language models (LLMs) have achieved impressive success in text-formatted learning problems, and most popular LLMs have been deployed in a black-box fashion. Meanwhile, fine-tuning is usually necessary for a specific downstream task to obtain better performance, and this functionality is provided by the owners of the black-box LLMs. To fine-tune a black-box LLM, labeled data are always required to adjust the model parameters. However, in many real-world applications, LLMs can label textual datasets with even better quality than skilled human annotators, motivating us to explore the possibility of fine-tuning black-box LLMs with unlabeled data. In this paper, we propose unsupervised prompt learning for classification with black-box LLMs, where the learning parameters are the prompt itself and the pseudo labels of unlabeled data. Specifically, the prompt is modeled as a sequence of discrete tokens, and every token has its own to-be-learned categorical distribution. On the other hand, for learning the pseudo labels, we are the first to consider the in-context learning (ICL) capabilities of LLMs: we first identify reliable pseudo-labeled data using the LLM, and then assign pseudo labels to other unlabeled data based on the prompt, allowing the pseudo-labeled data to serve as in-context demonstrations alongside the prompt. Those in-context demonstrations matter: previously, they are involved when the prompt is used for prediction while they are not involved when the prompt is trained; thus, taking them into account during training makes the prompt-learning and prompt-using stages more consistent. Experiments on benchmark datasets show the effectiveness of our proposed algorithm. After unsupervised prompt learning, we can use the pseudo-labeled dataset for further fine-tuning by the owners of the black-box LLMs.
BibTeX
@misc{zhang2024unsupervised,
  author = {Zhen-Yu Zhang and Jiandong Zhang and Huaxiu Yao and Gang Niu and Masashi Sugiyama},
  title = {{On Unsupervised Prompt Learning for Classification with Black-box Language Models}},
  year = {2024},
  eprint = {2410.03124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03124},
}
Figure