TL;DR
Zero-shot classification with Large Language Models (LLMs) typically lacks a modifiable decision boundary, making it difficult to tune for specific precision/recall needs or calculate threshold-independent metrics like AUROC. This paper proposes a "tournament" method that uses LLMs to perform pairwise comparisons between data instances, using the Elo rating system to generate a confidence ranking that significantly improves classification performance and enables standard evaluation metrics.
What Problem Does This Paper Address?
In traditional supervised learning, classifiers output a probability score. This allows practitioners to set a "decision threshold"—for example, requiring a 90% confidence score to flag a rare disease (prioritizing precision) or only 10% to ensure no cases are missed in a screening (prioritizing recall).
Zero-shot LLMs, however, usually provide a discrete "Yes" or "No" answer. This makes them "black boxes" regarding their internal confidence, preventing the calculation of the Area Under the Receiver Operating Characteristic curve (AUROC), a standard metric for comparing models across all possible thresholds. While "prompt steering" (telling a model to be more cautious) is a common workaround, the authors find it largely ineffective for fine-tuning the precision-recall trade-off.
Methods
The authors transform the classification task into a ranking task using a tournament structure:
- Pairwise Comparison: Instead of asking the LLM if a single instance belongs to a class, they present two instances and ask: "Which of these is more likely to be [Class X]?"
- Elo Rating System: Borrowing from competitive chess, every instance in the dataset starts with a base rating. When the LLM "picks" one instance over another, the winner’s rating increases and the loser’s decreases based on the expected outcome.
- Scheduling Strategies: To minimize the number of expensive LLM calls, the authors tested three ways to pair instances:
- Random: Simple random pairings.
- Graph-based: Pairing instances that have the fewest "connections" (matches) between them to reduce uncertainty.
- Swiss System: A common tournament format that pairs instances with similar current Elo ratings to refine local ordering.
- Scoring: After several rounds, the final Elo ratings provide a continuous ranking, which is then used to calculate AUROC and AUPRC.
Datasets
The method was evaluated across three distinct domains:
- CoLA (Corpus of Linguistic Acceptability): Determining if sentences are grammatically correct (General domain).
- CliniFact: Verifying if scientific abstracts support specific clinical claims (Biomedical literature).
- LCD Bench (Summary): Predicting 30-day mortality from summarized patient discharge notes (Clinical domain, derived from MIMIC-IV).
Key Insights and Ideas
- Ranking > Labeling: Forcing a model to choose between two examples extracts more nuanced signal than asking for a binary label. This "relative" judgment appears easier for the LLM than making an "absolute" judgment.
- Performance Boost: Surprisingly, the tournament method doesn't just provide a ranking; it actually improves the final classification accuracy (F1 score) compared to standard zero-shot labeling.
- Efficiency vs. Accuracy: Performance tends to plateau after 10–20 rounds. While more rounds provide a better ranking, the "Random" scheduling strategy proved remarkably robust, often matching more complex graph-based methods in early rounds.
- Calibration for Black Boxes: This method works entirely on text generations. Unlike other uncertainty estimation methods, it doesn't require access to the model's internal log-probabilities, making it ideal for closed-API models like GPT-4o.
Key Results
The tournament method consistently outperformed standard zero-shot and self-consistency baselines across all models (Llama 3.2 3B, GPT-4o-mini, Qwen 2.5, etc.).
- F1 Score Gains: On the CoLA dataset, Llama 3.2 3B’s F1 score jumped from 0.671 (standard zero-shot) to 0.866 using the tournament method.
- AUROC Improvements: As shown in Figure 1, the AUROC improves steadily as the number of tournament rounds increases, far exceeding the performance of simple zero-shot averages.
- Clinical Utility: On the LCD Bench mortality prediction task, the tournament method allowed for a clear ranking of patient risk, which is vital for clinical decision support where thresholds must be adjustable.
Limitations and Future Work
The primary drawback is inference cost. A tournament requires multiple passes over the dataset (N instances per round), which is significantly more expensive than a single zero-shot pass. The authors note that while performance asymptotes quickly, the optimal number of rounds likely scales with dataset size.
Future work includes exploring how this method scales to much larger datasets and whether it can be adapted for multi-class classification rather than just binary tasks. Additionally, the authors suggest investigating why certain models (like Phi-tiny-MoE) occasionally produced rankings that were inversely correlated with the truth (AUROC < 0.5) on specific tasks.
Summary updated