Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation

Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li

Introduction

Evaluating Large Language Models (LLMs) on massive benchmarks has become a significant bottleneck in AI development. As models and benchmarks scale, the cost of a single evaluation run can reach thousands of dollars in API fees or thousands of GPU hours. To mitigate these costs, researchers often employ "coresets"—small, representative subsets of a benchmark (e.g., 50 questions out of 5,000) intended to approximate the performance of the whole.

However, traditional coreset methods typically rely on a "one-size-fits-all" approach. They assume that a subset derived from the performance patterns of older, publicly available "source models" will remain representative for any new "target model." In this paper, we introduce TailoredBench, a dynamic evaluation framework that moves away from static subsets by constructing a customized coreset for every specific model under evaluation.

The Challenge: Prediction Inconsistency

Existing coreset selection methods rely on a "source-to-target" assumption: the belief that the performance patterns of source models can accurately predict which questions are representative for a new target model. Our research demonstrates that this assumption is often flawed.

If a target model possesses a different architecture or training data than the source models, the chosen subset loses its representativeness. This "prediction inconsistency" leads to highly inaccurate performance estimates. t-SNE visualizations of model predictions reveal that "representative" points (cluster centroids) derived from source models fail to represent the data distribution when viewed through the lens of a new target model. This consistency gap is the primary driver of error in static evaluation methods.

The TailoredBench Framework

TailoredBench addresses the inconsistency issue by adapting to the unique "personality" of the model being tested through a four-step pipeline:

  1. Global-coreset (G-set) Probing: The target model is first evaluated on a tiny, static "probe" set (typically just 10 examples). This provides an initial snapshot of the model's behavior.
  2. Adaptive Source Model Selection: Based on the G-set results, TailoredBench identifies a "Native Source Model Set"—a group of existing models that exhibit the highest prediction consistency with the target model. We found that models often show "intra-family affinity," where a model like Llama-3 aligns more closely with Llama-2 than with unrelated architectures like Mistral.
  3. Native-coreset (N-set) Construction: Using the performance data from these "kin" models, the system employs a scalable K-Medoids clustering algorithm to select a customized subset of examples (the N-set). This set is specifically designed to represent the benchmark's difficulty and diversity for that specific target model.
  4. Calibrated Performance Estimation: Rather than simply averaging the N-set results, TailoredBench applies a calibration factor derived from the native source models. This "nudges" the estimate to account for subtle variations between the subset and the full benchmark.

Experimental Validation

We validated TailoredBench across five diverse benchmarks—ARC Challenge, Hellaswag, GSM8K, Winogrande, and POPE—using evaluation results from over 300 different models.

Accuracy and Precision

TailoredBench achieved an average 31.4% reduction in Mean Absolute Error (MAE) in accuracy estimates compared to top-performing baselines like Anchor Points or gp-IRT. With an inference budget of only 20–40 examples, the method provides performance estimates remarkably close to testing on the full dataset of thousands of examples.

Ranking Reliability

Beyond raw accuracy, TailoredBench maintains high ranking reliability (measured by Kendall’s $\tau$). This ensures that the method doesn't just guess the score correctly but also places models in the correct leaderboard order, which is critical for comparative research.

Robustness and Efficiency

The method demonstrated significantly lower variance across randomized trials compared to random sampling or static coresets. Furthermore, our analysis found that element-wise distance (Manhattan distance) is more effective than standard correlation for measuring model similarity, as it better captures the discrete nature of "correct vs. incorrect" predictions.

Conclusion and Future Work

TailoredBench represents a shift toward more efficient, model-aware evaluation. By recognizing that different models "see" benchmarks differently, we can drastically reduce the computational cost of LLM development without sacrificing accuracy.

The primary limitation of this approach is its dependency on a "source pool" of existing evaluation results. While this is readily available for popular benchmarks with public leaderboards, applying TailoredBench to entirely new or niche benchmarks remains a challenge. Future work will explore minimizing initial probing costs and extending the framework to generative tasks where correctness is harder to define than in multiple-choice formats.

Summary updated

Abstract

Evaluating models on large benchmarks is very resource-intensive, especially during the period of rapid model evolution. Existing efficient evaluation methods estimate the performance of target models by testing them only on a small and static coreset of the benchmark, which is derived from the publicly available evaluation results of source models. These methods rely on the assumption that target models have high prediction consistency with source models. However, we demonstrate that it doesn't generalize well in practice. To alleviate the inconsistency issue, we present TailoredBench, a method that conducts customized evaluation tailored to each target model. Specifically, a Global-coreset is first constructed as a probe to identify the most consistent source models for each target model with an adaptive source model selection strategy. Afterwards, a scalable K-Medoids clustering algorithm is proposed to extend the Global-coreset to a tailored Native-coreset for each target model. According to the predictions on Native-coresets, we obtain the performance of target models on the whole benchmark with a calibrated estimation strategy. Comprehensive experiments on 5 benchmarks across over 300 models demonstrate that compared to best performing baselines, TailoredBench achieves an average reduction of 31.4% in MAE of accuracy estimates under the same inference budgets, showcasing strong effectiveness and generalizability.
BibTeX
@inproceedings{yuan2025beyond,
  author = {Peiwen Yuan and Yueqi Zhang and Shaoxiong Feng and Yiwei Li and Xinglin Wang and Jiayi Shi and Chuyi Tan and Boyuan Pan and Yao Hu and Kan Li},
  title = {{Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=bkg89PXN3Q},
}
Figure