Introduction
Large Language Models (LLMs) are highly sensitive to prompt engineering; even minor phrasing adjustments can significantly impact performance. While automated "Prompt-as-Optimizer" techniques like OPRO have emerged to address this, they typically rely on "gold labels"—manually verified correct answers—to score and iterate on candidate prompts. This dependency creates a bottleneck for novel tasks or private datasets where ground-truth answers are unavailable.
GLaPE (Gold Label-agnostic Prompt Evaluation) introduces a framework to optimize prompts without requiring any manually annotated data. By leveraging the model's internal consistency and consensus across different prompts, GLaPE identifies high-performing prompts that rival those optimized using supervised methods.
The GLaPE Framework
GLaPE replaces traditional accuracy metrics with a two-pillar unsupervised approach: Self-Consistency (SC) and Mutual-Consistency (MC) Refinement.
1. Self-Consistency (SC) Evaluation
SC is based on the principle that correct reasoning paths often converge on the same answer. GLaPE samples the model multiple times for a single prompt. A prompt that consistently yields the same answer across these samples receives a high SC score.
2. Mutual-Consistency (MC) Refinement
A known limitation of SC is that models can be "consistently wrong," producing incorrect answers with high confidence. To mitigate this, GLaPE evaluates consensus across multiple different prompts. If several prompts converge on "Answer X" while one outlier prompt yields "Answer Y," the outlier is penalized. This refinement process "pulls" individual prompt scores toward the collective consensus, effectively filtering out high-confidence hallucinations.
The final GLaPE score is a weighted balance of SC and MC, which is then used in a meta-prompting loop. In this loop, an LLM acts as an optimizer, generating new candidate prompts based on the highest-scoring previous iterations.
Experimental Setup and Results
The researchers evaluated GLaPE across eight diverse reasoning benchmarks, including arithmetic (GSM8K, MATH, MultiArith) and commonsense logic (StrategyQA, Big-Bench Date). Testing spanned various architectures, including GPT-3.5-turbo, Mistral-7B, Llama3-8B, and Gemma2-9B.
Alignment with Accuracy
A key finding is that GLaPE scores correlate much more strongly with actual task accuracy than raw SC scores. While SC can be misleading due to consistent errors, the mutual consistency check ensures that the metric tracks true performance.
Statistical analysis using Spearman correlation confirmed this. For instance, on the MultiArith dataset, the correlation with accuracy improved from 0.29 (using SC) to 0.88 (using GLaPE).
Performance Parity
GLaPE-optimized prompts achieved results comparable to—and sometimes exceeding—supervised methods. On the GSM8K benchmark, GLaPE reached 77.7% accuracy, slightly outperforming the gold-label-dependent OPRO method (76.6%). Across most datasets, GLaPE remained within 1-2% of the performance of prompts optimized with ground-truth labels.
Discussion and Limitations
GLaPE demonstrates that "consistency-as-truth" is a powerful heuristic in modern LLMs. However, the method is not without limitations. It struggles when a question is so difficult that the model provides the same incorrect answer across all prompts due to intrinsic knowledge errors. Additionally, the requirement for multiple samplings per prompt increases computational costs compared to single-pass evaluations.
Despite these challenges, GLaPE provides a robust path forward for prompt optimization in label-scarce environments, proving that model consensus can serve as a reliable proxy for truth.
Summary updated