Prompt-to-Leaderboard

Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N. Angelopoulos, Ion StoicaarXiv 2025 · DOI · arXiv

Abstract

Traditional Large Language Model (LLM) evaluations rely on static, aggregate scores that fail to capture the nuances of model performance across diverse tasks. We introduce Prompt-to-Leaderboard (P2L), a framework that trains models to predict a unique leaderboard for every individual prompt. By dynamically selecting the optimal model for a specific query, a P2L-based router achieved the top position on the Chatbot Arena leaderboard in January 2025. Our findings demonstrate that the ability to evaluate LLMs follows predictable scaling laws, improving consistently with model size and training data.

Introduction

Current LLM benchmarks, such as MMLU or the Chatbot Arena Elo ratings, provide a single numerical value to represent a model's overall capability. However, these averages obscure significant performance variance; a model optimized for mathematical reasoning may underperform in creative writing, yet both are collapsed into a single "average" score.

To address this, researchers often attempt to stratify data into sub-categories. However, obtaining a stable ranking for niche tasks requires a prohibitive amount of human preference data. Prompt-to-Leaderboard (P2L) addresses this challenge by enabling granular, prompt-level evaluation without requiring an infinite stream of new human labels for every possible use case.

The P2L Framework

The core innovation of P2L is the transformation of an LLM into an evaluator that predicts Bradley-Terry (BT) coefficients for a given input. Instead of generating text, the model outputs a vector representing a custom leaderboard tailored to the specific nuances of the prompt.

Architecture and Training

The P2L architecture utilizes pre-trained, instruction-tuned LLMs (ranging from 135M to 7B parameters). The standard language modeling head is replaced with a specialized "coefficient head" that outputs $M$ values, corresponding to the $M$ models under evaluation.

Training is conducted using human preference data—specifically 1.5 million crowdsourced pairs from Chatbot Arena. The model is optimized using a Bradley-Terry objective, learning to predict which model a human would likely prefer based on the specific requirements of the input prompt.

The P2L pipeline: from prompt input to a predicted leaderboard and downstream applications like routing and analysis.

Beyond Binary Preferences

The framework is further generalized into Prompt-to-Regression (P2R). By employing a "grounded Rao-Kupper model," P2L can account for "Ties" and "Both Bad" outcomes. This allows the system to measure absolute model reliability and quality rather than just relative preference.

Experimental Setup

The researchers evaluated P2L using two primary datasets:

  1. Chatbot Arena: 1.5 million human preference pairs covering 130 models, used for primary training.
  2. LiveBench: A verifiable, objective benchmark with 1,000 questions (math, coding, reasoning) used to test out-of-distribution generalization.

Results and Scaling Laws

The empirical evaluation of P2L reveals several key insights into the nature of LLM evaluation.

Scaling Laws for Evaluation

A significant finding is that the ability to evaluate LLMs follows power laws similar to the models themselves. As the parameter count of the P2L model increases and more training data is consumed, the accuracy of the predicted leaderboards improves consistently.

Validation loss and mean squared error showing that P2L performance scales with both model size and the number of training points seen.

Performance on Chatbot Arena

The P2L router demonstrated state-of-the-art performance. In January 2025, it secured the #1 spot on the Chatbot Arena leaderboard, outperforming the previous leader, Gemini-exp-1206, by 25 points. This success stems from the "Specialist Advantage": the router identifies when a smaller, more efficient model is actually superior for a specific task, rather than always defaulting to the largest available model.

Cost Efficiency and Analysis

P2L enables a superior cost-performance trade-off. By routing simpler prompts to cheaper models and reserving high-parameter models for complex queries, the P2L-7B router dominates the Pareto frontier. Furthermore, P2L allows for unsupervised strength/weakness analysis. By feeding unlabeled prompts into the model, researchers can automatically identify clusters where smaller models (like GPT-4o-mini) rival their larger counterparts.

Conclusion and Future Work

Prompt-to-Leaderboard represents a shift from static evaluation to dynamic, context-aware assessment. While the current implementation relies on the transitivity assumptions of the Bradley-Terry model and requires significant initial human data, it provides a powerful tool for both model deployment and analysis. Future research will focus on handling non-transitive human preferences and optimizing for specific performance guarantees under strict cost constraints.

Summary updated

Abstract

Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt. The core idea is to train an LLM taking natural language prompts as input to output a vector of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependent leaderboards allow for unsupervised task-specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce prompt-specific evaluations follows a power law scaling similar to that observed in LLMs themselves. In January 2025, the router we trained based on this methodology achieved the #1 spot on the Chatbot Arena leaderboard. Our code is available on GitHub at https://github.com/lmarena/p2l.
BibTeX
@misc{frick2025prompt,
  author = {Evan Frick and Connor Chen and Joseph Tennyson and Tianle Li and Wei-Lin Chiang and Anastasios N. Angelopoulos and Ion Stoica},
  title = {{Prompt-to-Leaderboard}},
  year = {2025},
  eprint = {2502.14855},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.14855},
}
Figure