Enabling Weak LLMs to Judge Response Reliability via Meta Ranking

Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang LiuSubmission to ICLR 2025 · Rejected

TL;DR

Large, expensive models like GPT-4 are typically required to reliably judge whether a language model's output is correct. Meta Ranking (MR) circumvents this by enabling lightweight models (like the 2.7B-parameter Phi-2) to assess response reliability through pairwise, cross-query comparisons against a tiny set of reference examples. This training-free approach matches or outperforms much larger evaluators in error detection and translates into practical gains for both model cascading and data-efficient instruction tuning.

What Problem Does This Paper Address?

Hallucinations and subtle reasoning errors remain ubiquitous in LLM outputs. While strong proprietary models can act as capable judges ("LLM-as-a-Judge") or score response uncertainty, they are expensive, slow, and cannot be deployed in secure local environments.

Conversely, small, local LLMs struggle at direct evaluation (e.g., standard few-shot prompting or asking $P(\text{True})$). They suffer from low in-context learning capacity, severe overconfidence in their own judgments, and sensitivity to prompt example order. The authors ask: Can we empower weak, lightweight LLMs to accurately judge response reliability without expensive fine-tuning or massive calibration datasets?

Methods

Rather than evaluating a single query-response pair $(Q_t, R_t)$ in isolation, Meta Ranking (MR) reframes evaluation into a series of pairwise comparisons across different queries:

  1. Reference Set: A static set of $N$ reference query-response pairs $(Q_i, R_i)$ with known reliability labels $S_i \in \{+1, -1\}$ (e.g., as few as 5 examples from a dev set) is maintained.
  2. Cross-Query Comparison: The weak judge LLM is prompted to determine whether $(Q_t, R_t)$ is more reliable, equally reliable, or less reliable than $(Q_i, R_i)$, yielding $r_i \in \{+1, 0, -1\}$. By comparing responses across different queries from the same base generator, the judge avoids self-preference and overconfidence biases.
  3. Weighted Aggregation: Comparisons are aggregated via signed, weighted voting: $$s = \sum_{i=1}^N S_i \cdot \delta_{\text{sgn}(S_i) \cdot r_i}$$ where $\delta$ weights the credibility of the outcome (e.g., ranking above a correct reference vs. ranking below an incorrect one). The target is deemed reliable if $s \ge 0$.

  4. Positive-KTO (pKTO): For instruction tuning, the authors introduce a variant of Kahneman-Tversky Optimization that applies an MSE loss to the reward of low-quality filtered data rather than pushing it away as entirely negative, preserving useful foundational knowledge while reinforcing high-quality patterns.

Datasets

The authors evaluate MR across multiple tasks and benchmarks (no new dataset was created):

  • Reasoning & Error Detection: 8 subsets of MMLU (STEM, Humanities, Social Sciences), 5 subsets of CMMLU (Chinese benchmarks), GSM8K (grade-school math), and JCommonsenseQA (Japanese reasoning).
  • Machine Translation: FLORES-200 (English $\leftrightarrow$ Chinese).
  • Code Translation: HumanEval-X (Python to Java).
  • Instruction Tuning: Alpaca-52k, evaluated on MT-Bench and AlpacaEval 2.0.

Key Insights and Ideas

  • Comparative judgment is easier than absolute calibration: Small models fail at producing calibrated absolute probabilities, but they excel at relative ranking—even when comparing two completely different queries.
  • Overconfidence mitigation: Presenting two separate query-response pairs forces the judge model to weigh relative logical coherence rather than defaulting to an overconfident "looks correct to me" stance.
  • Extreme sample efficiency: Traditional uncertainty calibration methods (e.g., Semantic Entropy, NumSemSets) demand hundreds of labeled samples to set decision thresholds. MR achieves high precision with as few as 1 to 5 reference pairs.
  • Curriculum-style data filtering: In SFT, dynamically filtering out lower-ranked samples between epochs using the evolving model itself outperforms one-off static data pruning.

Key Results

  • Error Detection: When evaluating LLaMA-2 responses on MMLU, MR using Phi-2 (2.7B) attained a precision of 0.77 (an absolute gain of +0.55 over its base task accuracy of 0.22). It outperformed $P(\text{True})$ with GPT-3.5-turbo (0.59) and approached GPT-4-turbo (0.89) while running nearly $3\times$ faster than standard few-shot prompts.
  • Model Cascading: Using Phi-2 + MR as a router to dispatch uncertain queries from open-source models (OpenChat-3.5 / Yi) to GPT-4-turbo matched GPT-4-turbo’s standalone reasoning performance while slashing API token consumption by 57% (0.43 relative tokens).
  • Instruction Tuning: Filtering Alpaca-52k iteratively via MR allowed fine-tuned Phi-2 (4.77 on MT-Bench) and LLaMA-7B (4.69 on MT-Bench) to surpass the baseline Alpaca-13B model (4.53), using roughly half the training tokens and cutting data-scoring time by 50% compared to Deita.

Limitations and Future Work

  • Low-Resource Languages: MR performance dropped on Japanese reasoning (precision 0.34) without task-specific fine-tuning, reflecting the underlying judge's limited multilingual capacity.
  • Strong LLMs Unexplored: The study focused on small models ($\le$7B); whether frontier models (e.g., GPT-4) benefit from MR remains an open question.
  • High-Stakes Reliability: Despite improvements, precision is not 100%, meaning mission-critical domains (medicine, legal, finance) still require human oversight.

Summary updated

Abstract

Despite the strong performance of large language models (LLMs) across a wide range of tasks, they still have reliability issues. Previous studies indicate that strong LLMs like GPT-4-turbo excel in evaluating the reliability of responses from LLMs, but face efficiency and local deployment issues. Thus, to enable weak LLMs to effectively assess the reliability of LLM responses, we propose a novel cross-query-comparison-based method called $\textit{Meta Ranking}$ (MR). Unlike previous few-shot methods that solely based on in-context learning capabilities in LLMs, MR assesses reliability by pairwise ranking the target query-response pair with multiple reference query-response pairs. We found that MR is highly effective in error detection for LLM responses, that MR with weaker LLMs, which have lower task performance, results in higher judgement precision against baselines with the same or even stronger models. Moreover, the method requires as few as five reference samples and significantly improving efficiency. We further demonstrate that MR can enhance strong LLMs' performance in two practical applications: model cascading and instruction tuning. In model cascading, we combine open- and closed-source LLMs to achieve performance comparable to GPT-4-turbo with lower costs. In instruction tuning, we use MR for iterative training data filtering, significantly reducing data processing time and enabling LLaMA-7B and Phi-2 to surpass 13B models with fewer training tokens. These results underscore the high potential of MR in both efficiency and effectiveness.
BibTeX
@misc{liu2024enabling,
  author = {Zijun Liu and Boqun Kou and Peng Li and Ming Yan and Ji Zhang and Fei Huang and Yang Liu},
  title = {{Enabling Weak LLMs to Judge Response Reliability via Meta Ranking}},
  year = {2024},
  eprint = {2402.12146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.12146},
}
Figure