TL;DR
This paper introduces a novel "tournament-based" method to calculate AUROC for zero-shot LLM classification. By framing classification as pairwise comparisons and using an Elo rating system, they enable LLMs to produce confidence scores, allowing for threshold-independent evaluation and improved classification performance compared to traditional zero-shot methods.
What Problem Does This Paper Address?
Traditional supervised classifiers output probability distributions, allowing users to adjust decision thresholds to balance precision and recall for different use cases. Zero-shot LLMs, however, typically provide a single discrete classification without an explicit decision boundary or confidence score. This makes it difficult to fairly compare them to supervised methods using threshold-independent metrics like AUROC, which are crucial for understanding classifier performance across various operating points. The paper aims to bridge this gap by enabling LLMs to generate comparable confidence scores for zero-shot tasks.
Methods
The core of the proposed method is to transform a binary classification task into a series of pairwise comparisons between instances. Instead of asking an LLM to classify an instance directly, the LLM is prompted to compare two instances and determine which one is more likely to belong to the positive class. These pairwise comparisons are then used to update an Elo rating for each instance, a system commonly used in competitive games like chess. Over many rounds of these "tournaments," the Elo ratings converge to reflect the relative confidence of the LLM for each instance. This induced ranking of instances can then be used to compute AUROC. The paper also investigates various "scheduling strategies" (Random, Graph, Swiss) for pairing instances in these tournaments, aiming to minimize the number of comparisons needed to achieve stable AUROC scores.
Datasets
The authors evaluated their method on three datasets:
- CoLA (Corpus of Linguistic Acceptability): A dataset of linguistic acceptability judgments.
- CliniFact: A dataset of scientific claims from clinical trial protocols, labeled for support by abstracts.
- LCD Bench (Summ): A summarized version of the Long Clinical Document Benchmark, focusing on predicting out-of-hospital mortality from patient discharge notes.
Key Insights and Ideas
- LLMs can be "tournament-trained" for AUROC: The core idea of using pairwise comparisons and Elo ratings to derive confidence scores for zero-shot LLMs is a novel way to enable threshold-independent evaluation.
- Pairwise comparisons unlock confidence scores: By forcing LLMs to make relative judgments, the method implicitly generates a ranking that can be translated into confidence scores, something direct zero-shot classification struggles with.
- Scheduling matters, but not drastically: While different scheduling strategies (Random, Graph, Swiss) were explored, the paper found that even a simple Random scheduler performed surprisingly well, and the differences between schedulers converged over rounds.
- Tournament approach improves F1 scores: Beyond just AUROC, the method led to significant improvements in F1 scores for both datasets and LLMs tested, suggesting the generated confidence scores are more robust than direct zero-shot outputs.
- Prompt steering for precision/recall is limited: The paper found that explicitly steering prompts to favor precision or recall had limited effectiveness, reinforcing the need for methods like theirs that can derive these trade-offs post-hoc.
Key Results
- The proposed tournament-based method consistently outperformed traditional zero-shot and self-consistency baselines in terms of AUROC across all datasets and LLMs tested.
- The method led to substantial improvements in F1 scores. For example, on CoLA, Llama improved from 0.671 to 0.866, and GPT-40-mini improved from 0.895 to 0.910. For CliniFact, Llama's F1 score increased from 0.502 to 0.651, and GPT-40-mini's from 0.701 to 0.753.
- GPT-40-mini generally outperformed Llama-3.2-3B-Instruct across tasks, as expected.
- The "Graph" scheduler showed promise in some settings, but the "Random" scheduler was also competitive.
Limitations and Future Work
- Dataset limitations: The work was evaluated on only three datasets, which may not fully represent the diversity of classification tasks. The selection of models, while representative, also doesn't cover all LLMs.
- Inference time: The tournament method requires multiple passes over the dataset (equivalent to multiple inference passes per instance), making it slower than simple zero-shot classification.
- Dataset size and rounds: The optimal number of rounds for the tournament might be dataset-size dependent. Future work could explore this relationship to determine feasibility for larger datasets.
- Model behavior: The paper noted an interesting case where one model (DeepSeek-R1-Distill-Qwen-1.5B) failed to follow prompt instructions, producing unexpected output. Another model (Phi-tiny-MoE) performed below chance on one task, possibly due to misinterpreting the task. These highlight areas for future investigation into model behavior and robustness.
Summary updated