League: Leaderboard Generation on Demand

Jian Wu, Jiayu Zhang, Dongyuan Li, Linyi Yang, Aoxiao Zhong, Renhe Jiang, Qingsong Wen, Yue Zhang

TL;DR

With thousands of academic papers published daily, keeping track of the state-of-the-art has become an impossible task for human researchers. LEAGUE (Leaderboard Auto Generation) is a novel framework that leverages Large Language Models (LLMs) to automatically crawl, extract, and dynamically generate up-to-date research leaderboards. By capturing not just scores but also experimental settings, LEAGUE ensures fair comparisons at a fraction of the time and cost of manual curation.

What Problem Does This Paper Address?

In fast-moving fields like Artificial Intelligence, staying abreast of the state-of-the-art (SOTA) is overwhelming. While platforms like Papers with Code are incredibly valuable, they rely heavily on manual submissions and curation. As a result, a massive gap has formed between the surge of paper submissions and the updates of corresponding leaderboards. For instance, the authors note that the popular HotpotQA multi-hop question-answering leaderboard has remained stagnant since 2023, despite dozens of newer methods being published.

Previous attempts to automate this process focused purely on Scientific Named Entity Recognition (NER)—extracting isolated tasks, datasets, and models. However, these approaches yield static snapshots and fail to address three critical challenges:

  1. Paper Coverage: Finding all relevant papers on a rapidly evolving topic.
  2. Fair Comparison: Ensuring models are compared under standardized evaluation settings (accounting for variations in model size, training data volume, and hyperparameters).
  3. Timeliness: Keeping the leaderboard continuously updated.

Methods

LEAGUE introduces a systematic, four-stage pipeline designed to build and maintain leaderboards end-to-end:

Figure 2: The League framework for leaderboard automatic generation. In Stage 1, we automatically crawl scientific papers from arXiv. In Stage 2, we retrieve, extract, and classify tables from the LaTeX code. In Stage 3, we select the main results tables and extract datasets, metrics, results, and experiment settings from the main results table. In Stage 4, we generate Leaderboards from the selected results and evaluate the quality.
  1. Paper Collection and Split: The system automatically crawls papers from arXiv and top-tier conferences based on a target topic. To prevent token bloat and stay within LLM context windows, it strips out unproductive LaTeX code and isolates the "Experiment" sections.
  2. Table Extraction and Classification: Using few-shot Chain-of-Thought (CoT) prompting, an LLM categorizes tables into "Main Results," "Ablation Study," or "Others," filtering out everything except the core performance tables.
  3. Table Unpacking and Integration: The framework extracts data into a structured quintuple: [Paper Title, Dataset, Metrics, Experiment Settings, Experiment Results]. Extracting experimental settings (like model size or data subsets) is the key innovation that enables fair baseline alignment.
  4. Leaderboard Generation and Evaluation: Candidate leaderboards are refined, deduplicated, and ranked by a third-party LLM. Crucially, the framework runs in an iterative loop (typically 5 iterations), using an LLM-as-a-judge to continuously score and optimize the leaderboard's quality.

Compared to prior work, LEAGUE stands out by being fully dynamic, multi-document capable, and aware of experimental settings:

Table 1: Comparison between related work and our approach. Data Source: Source of leader- boards. NProg.: NLP-progress, PwC: paperswithcode. Experiment Settings: whether the experi- ment settings are extracted as part of leaderboards. Multi Document: whether the leaderboards are constructed from multiple papers. Dynamic: whether the generated leaderboards can be updated dynamically.

Datasets

The authors evaluated LEAGUE by harvesting papers from open-access platforms and top-tier AI venues (including ACL, EMNLP, NeurIPS, ICML, ICLR, and NAACL). They focused on generating leaderboards for specific benchmarks, including:

  • HotpotQA: For multi-hop question answering.
  • LA Dataset: For semi-supervised medical image segmentation.
  • LIVE Dataset: For image quality assessment.

To validate the intermediate extraction steps (Table Classification and Table NER), the authors also manually annotated a validation dataset consisting of 354 tables from 72 papers (containing 172 methods, 31 datasets, 49 settings, and 84 metrics).

Key Insights and Ideas

  • Context Matters for Fairness: Simply extracting a number from a table isn't enough. LEAGUE demonstrates that capturing the experimental settings column is vital; otherwise, a small model might be unfairly penalized against a massive model, or a zero-shot method compared directly against a fully fine-tuned one.
  • LLMs are Reliable Judges of Structure: The paper reveals a strong positive correlation (up to 0.76 Pearson correlation using o1-preview) between human expert evaluations and LLM-generated quality scores, proving that automated evaluation can serve as a reliable proxy for human judgment.
  • The Power of Iteration: The authors found that looping the generation process up to 5 times drastically improved the logical structure and coverage of the leaderboards, as the model learned to refine missing items and discard redundant metrics over time.

Key Results

LEAGUE achieves performance remarkably close to manual human curation while offering a massive leap in efficiency:

Quantitative evaluation of leaderboard quality across different models and item lengths.
  • Speedup: Constructing a comprehensive 20-item leaderboard takes a human expert over 18 minutes. LEAGUE (using GPT-4o) accomplishes this in under 2 minutes—a 5x to 10x speedup.
  • Quality: For a 20-item leaderboard, LEAGUE achieved a 67.58% recall and 70.33% precision in topic-related quality. Its content quality scores closely mirrored human writing across metrics like coverage, structure, and timeliness.
  • Cost Efficiency: Generating a complete, 20-entry leaderboard costs less than $14 in API tokens, making it highly practical for continuous deployment.

Limitations and Future Work

Despite its success, the authors note a few limitations:

  • Retrieval Dependence: LEAGUE is inherently bound by the quality of its initial paper crawler. If the retrieval model misses a key paper during Stage 1, it will never make it to the leaderboard.
  • Metric Fragmentation: Different papers often use slightly different metrics or abbreviations to evaluate the same task. While LEAGUE attempts to align these, extreme variations or sparse data (where >60% of papers omit a metric) force the system to discard that metric entirely to maintain leaderboard clarity.

Future work will focus on developing more sophisticated retrieval models to maximize paper coverage and improving cross-paper metric normalization.

Summary updated

Abstract

This paper introduces Leaderboard Auto Generation (LAG), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence (AI). Faced with a large number of AI papers updated daily, it becomes difficult for researchers to track every paper's proposed methods, experimental results, and settings, prompting the need for efficient automatic leaderboard construction. While large language models (LLMs) offer promise in automating this process, challenges such as multi-document summarization, leaderboard generation, and experiment fair comparison still remain under exploration. LAG solves these challenges through a systematic approach that involves the paper collection, experiment results extraction and integration, leaderboard generation, and quality evaluation. Our contributions include a comprehensive solution to the leaderboard construction problem, a reliable evaluation method, and experimental results showing the high quality of leaderboards.
BibTeX
@misc{wu2025league,
  author = {Jian Wu and Jiayu Zhang and Dongyuan Li and Linyi Yang and Aoxiao Zhong and Renhe Jiang and Qingsong Wen and Yue Zhang},
  title = {{League: Leaderboard Generation on Demand}},
  year = {2025},
  eprint = {2502.18209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.18209},
}
Figure