Introduction
Evaluating Large Language Models (LLMs) on massive benchmarks has become a significant bottleneck in AI development. As models and benchmarks scale, the cost of a single evaluation run can reach thousands of dollars in API fees or thousands of GPU hours. To mitigate these costs, researchers often employ "coresets"—small, representative subsets of a benchmark (e.g., 50 questions out of 5,000) intended to approximate the performance of the whole.
However, traditional coreset methods typically rely on a "one-size-fits-all" approach. They assume that a subset derived from the performance patterns of older, publicly available "source models" will remain representative for any new "target model." In this paper, we introduce TailoredBench, a dynamic evaluation framework that moves away from static subsets by constructing a customized coreset for every specific model under evaluation.
The Challenge: Prediction Inconsistency
Existing coreset selection methods rely on a "source-to-target" assumption: the belief that the performance patterns of source models can accurately predict which questions are representative for a new target model. Our research demonstrates that this assumption is often flawed.
If a target model possesses a different architecture or training data than the source models, the chosen subset loses its representativeness. This "prediction inconsistency" leads to highly inaccurate performance estimates. t-SNE visualizations of model predictions reveal that "representative" points (cluster centroids) derived from source models fail to represent the data distribution when viewed through the lens of a new target model. This consistency gap is the primary driver of error in static evaluation methods.
The TailoredBench Framework
TailoredBench addresses the inconsistency issue by adapting to the unique "personality" of the model being tested through a four-step pipeline:
- Global-coreset (G-set) Probing: The target model is first evaluated on a tiny, static "probe" set (typically just 10 examples). This provides an initial snapshot of the model's behavior.
- Adaptive Source Model Selection: Based on the G-set results, TailoredBench identifies a "Native Source Model Set"—a group of existing models that exhibit the highest prediction consistency with the target model. We found that models often show "intra-family affinity," where a model like Llama-3 aligns more closely with Llama-2 than with unrelated architectures like Mistral.
- Native-coreset (N-set) Construction: Using the performance data from these "kin" models, the system employs a scalable K-Medoids clustering algorithm to select a customized subset of examples (the N-set). This set is specifically designed to represent the benchmark's difficulty and diversity for that specific target model.
- Calibrated Performance Estimation: Rather than simply averaging the N-set results, TailoredBench applies a calibration factor derived from the native source models. This "nudges" the estimate to account for subtle variations between the subset and the full benchmark.
Experimental Validation
We validated TailoredBench across five diverse benchmarks—ARC Challenge, Hellaswag, GSM8K, Winogrande, and POPE—using evaluation results from over 300 different models.
Accuracy and Precision
TailoredBench achieved an average 31.4% reduction in Mean Absolute Error (MAE) in accuracy estimates compared to top-performing baselines like Anchor Points or gp-IRT. With an inference budget of only 20–40 examples, the method provides performance estimates remarkably close to testing on the full dataset of thousands of examples.
Ranking Reliability
Beyond raw accuracy, TailoredBench maintains high ranking reliability (measured by Kendall’s $\tau$). This ensures that the method doesn't just guess the score correctly but also places models in the correct leaderboard order, which is critical for comparative research.
Robustness and Efficiency
The method demonstrated significantly lower variance across randomized trials compared to random sampling or static coresets. Furthermore, our analysis found that element-wise distance (Manhattan distance) is more effective than standard correlation for measuring model similarity, as it better captures the discrete nature of "correct vs. incorrect" predictions.
Conclusion and Future Work
TailoredBench represents a shift toward more efficient, model-aware evaluation. By recognizing that different models "see" benchmarks differently, we can drastically reduce the computational cost of LLM development without sacrificing accuracy.
The primary limitation of this approach is its dependency on a "source pool" of existing evaluation results. While this is readily available for popular benchmarks with public leaderboards, applying TailoredBench to entirely new or niche benchmarks remains a challenge. Future work will explore minimizing initial probing costs and extending the framework to generative tasks where correctness is harder to define than in multiple-choice formats.
Summary updated