TL;DR
Fine-tuning commercial, black-box LLMs usually requires manually labeled data—an expensive bottleneck. This paper introduces PPD (Prompt learning with Pseudo-labeled Demonstrations), an unsupervised prompt optimization framework that simultaneously learns discrete prompt tokens and discovers high-quality pseudo-labels for unlabeled datasets. By incorporating retrieved in-context demonstrations directly into the black-box training objective, PPD consistently outperforms zero-shot inference and existing prompt-tuning baselines on GLUE and MMLU benchmarks.
What Problem Does This Paper Address?
Top-tier large language models (LLMs) such as GPT-4 are predominantly deployed as black boxes: users interact with them via commercial APIs without access to model weights, latent activations, or backpropagation gradients. Adapting these models to downstream classification tasks typically demands labeled datasets for fine-tuning or prompt optimization.
While LLMs are surprisingly capable zero-shot annotators, using naive self-training (labeling data and immediately fine-tuning) risks severe overfitting on noisy labels. Furthermore, existing black-box prompt-tuning algorithms optimize prompts in isolation and only prepend few-shot demonstrations at inference time. This creates a fundamental disconnect between how prompts are trained and how they are actually deployed with in-context learning (ICL).
Methods
The authors introduce PPD, which optimizes a discrete prompt and pseudo-labels concurrently without requiring labeled data:
[Unlabeled Data]
│
▼
1. Confident Pseudo-Label Identification (Linguistic or Bias-Reduced Confidence)
│
▼
2. Dynamic In-Context Demonstration Construction (K-NN retrieval + prompt-assigned labels)
│
▼
3. Joint Prompt Optimization via Policy Gradients (VR-PGE) + Entropy Regularization
- Reliable Pseudo-Label Identification: Using zero-shot API outputs, the method calculates prediction confidence scores either via multi-temperature average linguistic confidence ($c^{LG}$) or bias-reduced confidence ($c^{RD}$, which normalizes against random-context priors). Instances exceeding a threshold ($\gamma = 0.7$) are designated as initial anchors.
- Dynamic In-Context Demonstrations: For any query sample, $K$-nearest neighbors are retrieved using sentence embeddings. The black-box model predicts their pseudo-labels on the fly under the current prompt, and these pairs serve as in-context demonstrations ($D$).
- Training Objective: The discrete prompt $z$ is optimized to ensure that when the model evaluates a reliable sample conditioned on the learned prompt and the pseudo-labeled demonstrations, its prediction matches the sample's reliable pseudo-label: $$\mathcal{L}_{\text{main}}(z) = \sum_{l=1}^n \mathbf{1}[c_l \ge \gamma] \cdot \ell\big(f(x_l, z, D_l),\, f(x_l, \emptyset, \emptyset)\big)$$
- Entropy Regularization & Optimization: Each token position in prompt $z$ is parameterized as a categorical distribution over a candidate vocabulary mined via Pointwise Mutual Information (PMI). To avoid flat, ambiguous token distributions at test time, an entropy minimization penalty ($\alpha \sum H(p_i)$) forces distributions to become sharp and deterministic. The discrete parameters are updated via a Variance-Reduced Policy Gradient Estimator (VR-PGE) and projected stochastic gradient descent.
Datasets
The authors evaluate PPD across two major benchmark suites:
- GLUE: CoLA, SST-2, QQP, MRPC, MNLI, WNLI, and RTE.
- MMLU (9 subsets): Astronomy (AST), High-School Computer Science (HSCS), High-School Mathematics (HSM), College Mathematics (CMath), College Computer Science (CCS), College Medicine (CMed), Management (MAN), Marketing (MAR), and All-Random (RND).
No new datasets are introduced; all evaluation uses standardized splits without ground-truth labels during prompt learning.
Key Insights and Ideas
- Bridging the Train-Test Gap in ICL: Standard prompt learning optimizes a prompt alone and later appends demonstrations at test time. PPD places pseudo-labeled demonstrations inside the prompt-learning loop, ensuring the prompt is optimized specifically to exploit demonstration context.
- Mutual Reinforcement: Prompt learning and pseudo-labeling form a virtuous cycle: better prompts yield more accurate in-context demonstration labels, which in turn steer the policy gradient updates toward better prompts.
- Entropy Penalization Prevents Prompt Degeneracy: Without minimizing Shannon entropy across token categorical distributions, learned distributions remain dispersed, leading to erratic prompt sampling during inference.
Key Results
Experiments conducted using OpenAI’s gpt-4 and gpt-4o-mini show that PPD consistently outperforms zero-shot prediction (Direct), standard pseudo-labeled few-shot inference (ICL), and adapted black-box prompt baselines (BDPL, RLPrompt):
- GLUE: On SST-2, PPD (RD) achieves 91.5% accuracy versus 89.5% for Direct and 90.2% for BDPL (RD). On MNLI, PPD reaches 92.1% (vs. 90.2% for ICL-RD).
- MMLU: On challenging reasoning tasks, PPD demonstrates substantial gains over Direct prompting:
- College Mathematics (CMath): 44.9% vs. 40.5% (Direct).
- College Computer Science (CCS): 72.8% vs. 68.2% (Direct).
- Management (MAN): 79.8% vs. 76.6% (Direct).
- Demonstration Scaling: Testing $K \in \{3, 5\}$ revealed that $K=5$ paired with cross-entropy loss yielded the most consistent improvements across tasks.
- Model Portability: Both
gpt-4and the lightergpt-4o-miniexhibited consistent accuracy gains when evaluated with PPD prompts over direct zero-shot prompting.
Limitations and Future Work
- API Query Costs: Estimating gradients via VR-PGE requires repeated sampling across batch iterations and multiple forward passes per prompt token, which can become expensive on proprietary APIs.
- Sensitivity to Zero-Shot Baselines: The framework relies on the underlying LLM having sufficient baseline capability to yield well-calibrated confidence scores. If initial predictions on an obscure task are poor, noise may propagate into the demonstration pool.
- Downstream Fine-Tuning Integration: While the authors position the generated pseudo-labeled dataset as an asset for subsequent API fine-tuning, the paper primarily evaluates inference-time prompt accuracy rather than post-fine-tuning downstream checkpoints.
Summary updated