Abstract
Traditional Large Language Model (LLM) evaluations rely on static, aggregate scores that fail to capture the nuances of model performance across diverse tasks. We introduce Prompt-to-Leaderboard (P2L), a framework that trains models to predict a unique leaderboard for every individual prompt. By dynamically selecting the optimal model for a specific query, a P2L-based router achieved the top position on the Chatbot Arena leaderboard in January 2025. Our findings demonstrate that the ability to evaluate LLMs follows predictable scaling laws, improving consistently with model size and training data.
Introduction
Current LLM benchmarks, such as MMLU or the Chatbot Arena Elo ratings, provide a single numerical value to represent a model's overall capability. However, these averages obscure significant performance variance; a model optimized for mathematical reasoning may underperform in creative writing, yet both are collapsed into a single "average" score.
To address this, researchers often attempt to stratify data into sub-categories. However, obtaining a stable ranking for niche tasks requires a prohibitive amount of human preference data. Prompt-to-Leaderboard (P2L) addresses this challenge by enabling granular, prompt-level evaluation without requiring an infinite stream of new human labels for every possible use case.
The P2L Framework
The core innovation of P2L is the transformation of an LLM into an evaluator that predicts Bradley-Terry (BT) coefficients for a given input. Instead of generating text, the model outputs a vector representing a custom leaderboard tailored to the specific nuances of the prompt.
Architecture and Training
The P2L architecture utilizes pre-trained, instruction-tuned LLMs (ranging from 135M to 7B parameters). The standard language modeling head is replaced with a specialized "coefficient head" that outputs $M$ values, corresponding to the $M$ models under evaluation.
Training is conducted using human preference data—specifically 1.5 million crowdsourced pairs from Chatbot Arena. The model is optimized using a Bradley-Terry objective, learning to predict which model a human would likely prefer based on the specific requirements of the input prompt.
Beyond Binary Preferences
The framework is further generalized into Prompt-to-Regression (P2R). By employing a "grounded Rao-Kupper model," P2L can account for "Ties" and "Both Bad" outcomes. This allows the system to measure absolute model reliability and quality rather than just relative preference.
Experimental Setup
The researchers evaluated P2L using two primary datasets:
- Chatbot Arena: 1.5 million human preference pairs covering 130 models, used for primary training.
- LiveBench: A verifiable, objective benchmark with 1,000 questions (math, coding, reasoning) used to test out-of-distribution generalization.
Results and Scaling Laws
The empirical evaluation of P2L reveals several key insights into the nature of LLM evaluation.
Scaling Laws for Evaluation
A significant finding is that the ability to evaluate LLMs follows power laws similar to the models themselves. As the parameter count of the P2L model increases and more training data is consumed, the accuracy of the predicted leaderboards improves consistently.
Performance on Chatbot Arena
The P2L router demonstrated state-of-the-art performance. In January 2025, it secured the #1 spot on the Chatbot Arena leaderboard, outperforming the previous leader, Gemini-exp-1206, by 25 points. This success stems from the "Specialist Advantage": the router identifies when a smaller, more efficient model is actually superior for a specific task, rather than always defaulting to the largest available model.
Cost Efficiency and Analysis
P2L enables a superior cost-performance trade-off. By routing simpler prompts to cheaper models and reserving high-parameter models for complex queries, the P2L-7B router dominates the Pareto frontier. Furthermore, P2L allows for unsupervised strength/weakness analysis. By feeding unlabeled prompts into the model, researchers can automatically identify clusters where smaller models (like GPT-4o-mini) rival their larger counterparts.
Conclusion and Future Work
Prompt-to-Leaderboard represents a shift from static evaluation to dynamic, context-aware assessment. While the current implementation relies on the transitivity assumptions of the Bradley-Terry model and requires significant initial human data, it provides a powerful tool for both model deployment and analysis. Future research will focus on handling non-transitive human preferences and optimizing for specific performance guarantees under strict cost constraints.
Summary updated