From Rankings to Insights: Evaluation Should Shift Focus from Leaderboard to Feedback

Zongqi Wang, Tianle Gu, Chen Gong, Xin Tian, Siqi Bao, Yujiu Yang

Introduction

The current landscape of Large Language Model (LLM) evaluation is dominated by aggregate metrics and leaderboards. While platforms like Chatbot Arena provide valuable high-level rankings through ELO scores, these single-number summaries are often opaque for developers and power users. A high-ranking model may excel in general conversation but fail significantly in specific domains like low-level systems programming or nuanced creative writing. To address this lack of granularity, we present the SCAN framework, which shifts the focus from simple rankings to detailed, actionable feedback.

The SCAN Framework

Structured Capability Assessment and Navigation (SCAN) is a comprehensive evaluation system designed to map the vast landscape of LLM capabilities. It moves beyond static benchmarks by employing a dynamic approach to taxonomy generation and query synthesis, coupled with a novel evaluation methodology.

Overview of the SCAN framework, illustrating the pipeline from taxonomy construction to evaluation and analysis.

Technical Pillars

The SCAN framework rests on four primary technical pillars:

  1. TaxBuilder: Rather than relying on manually curated categories, TaxBuilder utilizes an LLM to automatically extract "capability-indicating tags" from millions of real-world user queries. These tags are organized into a hierarchical tree using a recursive node-insertion process. This ensures the taxonomy is both scalable and capable of evolving alongside new AI use cases.
  2. RealMix: To populate the thousands of fine-grained categories in the taxonomy, RealMix synthesizes realistic evaluation queries. By mixing content from multiple real user prompts, it generates high-quality, diverse data that avoids the repetitive or "robotic" nature of purely synthetic datasets.
  3. PC² (Pre-Comparison-derived Criteria): Traditional "LLM-as-a-Judge" methods often face a trade-off between the speed of pointwise scoring and the accuracy of pairwise comparisons. PC² bridges this gap by having a judge model first analyze responses from multiple models to extract specific, weighted evaluation criteria. These custom criteria are then used to provide accurate, individual scores for each model.
  4. Visualization and Analysis Toolkits: SCAN includes a suite of diagnostic tools, such as the "Failure Mode Explorer," which allows researchers to navigate complex datasets and pinpoint specific areas of model weakness.

Experimental Results and Insights

The application of SCAN to current state-of-the-art models reveals that performance is rarely uniform across all domains. This "spiky" capability profile is often hidden by aggregate scores. For instance, while a model like GPT-OSS-120B may rank at the top of a general leaderboard, SCAN diagnostics reveal specific relative weaknesses in domains such as roleplay or bioengineering.

Granular performance comparison across programming languages, demonstrating the 'spiky' capability profiles of various LLMs.

As shown in the granular coding analysis, models that appear similar in aggregate performance often exhibit significant variance across different programming languages. A model might be a leader in JavaScript but lag behind competitors in C or Java. This level of detail is critical for developers who need to select the right model for a specific technical stack.

Furthermore, the PC² judging method demonstrated superior alignment with human intuition, achieving an accuracy of up to 69.6% compared to the 56.9% accuracy of naive pointwise judging. This suggests that providing the judge model with comparative context before scoring leads to much more reliable automated evaluations.

Conclusion and Future Work

SCAN represents a necessary shift in LLM evaluation, moving from competitive rankings to diagnostic insights. By providing a structured way to understand where models excel and where they falter, SCAN enables more targeted model development and more informed deployment decisions. Future iterations of the framework, tentatively titled "SCAN-Anything," aim to expand these diagnostic capabilities into multimodal domains, including image, audio, and video processing, while also integrating deeper assessments of safety and ethical alignment.

Summary updated

BibTeX
@inproceedings{wang2026from,
  author = {Zongqi Wang and Tianle Gu and Chen Gong and Xin Tian and Siqi Bao and Yujiu Yang},
  title = {{From Rankings to Insights: Evaluation Should Shift Focus from Leaderboard to Feedback}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.126},
}
Figure