LAG: LLM agents for Leaderboard Auto Generation on Demanding

Jian Wu, Jiayu Zhang, Dongyuan Li, Linyi Yang, Aoxiao Zhong, Renhe Jiang, Qingsong Wen, Yue Zhang

Abstract

This paper introduces Leaderboard Auto Generation (LAG), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence (AI). Faced with a large number of AI papers updated daily, it becomes difficult for researchers to track every paper's proposed methods, experimental results, and settings, prompting the need for efficient automatic leaderboard construction. While large language models (LLMs) offer promise in automating this process, challenges such as multi-document summarization, leaderboard generation, and experiment fair comparison still remain under exploration. LAG solves these challenges through a systematic approach that involves the paper collection, experiment results extraction and integration, leaderboard generation, and quality evaluation. Our contributions include a comprehensive solution to the leaderboard construction problem, a reliable evaluation method, and experimental results showing the high quality of leaderboards.

The Stagnation Gap in AI Research

The pace of AI research is currently overwhelming. In 2025 alone, LLM-related submissions to arXiv exceeded 55,000, yet manual leaderboard platforms like Papers-With-Code only saw a fraction of that growth. This creates a "stagnation gap" where official leaderboards often feature outdated methods while the true state-of-the-art (SOTA) is buried in a recent PDF.

The growing gap between arXiv paper submissions and manual leaderboard updates.

Existing automated tools for scientific information extraction typically focus on simple Named Entity Recognition (NER)—identifying a model name or a score. However, they fail at the "fair comparison" problem: they don't account for varying experimental settings (like model size, dataset versions, or hyperparameters) that make a raw score comparison meaningful. LEAGUE aims to bridge this gap by providing a systematic, end-to-end pipeline for Leaderboard Generation on Demand.

The LEAGUE Framework

LEAGUE operates through a four-stage iterative pipeline designed to ensure high-quality, fair comparisons across multiple documents.

The four-stage iterative pipeline of the LEAGUE framework.
  1. Paper Collection and Split: The system crawls arXiv and top-tier conferences (ACL, NeurIPS, etc.) for a specific topic. It filters papers by date and relevance, then splits the LaTeX/PDF content into manageable sections, focusing on the "Experiment" blocks.
  2. Table Extraction and Classification: Using a few-shot Chain-of-Thought (CoT) prompting approach, LLMs identify "Main Results" tables while discarding ablation studies or auxiliary data tables.
  3. Table Unpacking and Integration: The framework extracts "quintuples" from the tables: Paper Title, Dataset, Metric, Experiment Settings, and Results. Extracting the Experiment Settings is the standard-setting feature here, as it allows the system to align results under comparable conditions.
  4. Generation and Evaluation: The quintuples are refined and ranked to form a candidate leaderboard. LEAGUE then employs an "LLM-as-a-Judge" mechanism to score the leaderboard based on four criteria: Coverage, Timeliness (Latest), Logical Structure, and Multiaspect quality.

The entire process is iterated (typically five times) to refine the results and ensure the highest-quality papers are included.

Evaluation and Datasets

The authors evaluated LEAGUE across several diverse research domains to prove its versatility:

  • Multi-hop Question Answering: Specifically using the HotpotQA dataset.
  • Medical Imaging: Semi-supervised medical image segmentation on the LA dataset.
  • Image Quality Assessment (IQA): Evaluated on the LIVE dataset.
  • Internal Validation: The authors manually annotated 354 tables from 72 papers to create a "gold standard" for testing LEAGUE’s table classification and entity extraction accuracy.

Key Results and Insights

LEAGUE demonstrates human-level quality and massive speedups in leaderboard construction:

  • Performance: In a 20-item leaderboard test, LEAGUE achieved a "Coverage" score of 4.12 and a "Structure" score of 4.16 (out of 5), closely approaching human performance (4.72 and 4.34, respectively).
  • Efficiency: Generating a comprehensive 20-entry leaderboard took LEAGUE approximately 2 minutes, compared to over 18 minutes for a human expert.
  • Cost Efficiency: Using GPT-4o, the computational cost to generate a full leaderboard was only about $2.18, making it a highly scalable solution for research labs or individual students.
  • Fairness via Metadata: A leaderboard isn't just a list of numbers; it's a list of numbers contextualized by experimental settings. By forcing the LLM to extract training strategies and model sizes, LEAGUE prevents "unfair" rankings.
  • LLM-as-Judge Alignment: The researchers found that the O1-preview model's evaluation of leaderboard quality had a high Pearson correlation (0.76) with human experts, suggesting that quality control can be largely automated.

Conclusion and Future Work

While LEAGUE offers a robust solution for automated tracking of SOTA, challenges remain in retrieval quality and metric inconsistency. Future work will explore more sophisticated retrieval models and expand the framework to handle multi-modal papers where results are primarily presented in charts rather than tables.

Summary updated

Abstract

This paper introduces Leaderboard Auto Generation (LAG), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence (AI). Faced with a large number of AI papers updated daily, it becomes difficult for researchers to track every paper's proposed methods, experimental results, and settings, prompting the need for efficient automatic leaderboard construction. While large language models (LLMs) offer promise in automating this process, challenges such as multi-document summarization, leaderboard generation, and experiment fair comparison still remain under exploration. LAG solves these challenges through a systematic approach that involves the paper collection, experiment results extraction and integration, leaderboard generation, and quality evaluation. Our contributions include a comprehensive solution to the leaderboard construction problem, a reliable evaluation method, and experimental results showing the high quality of leaderboards.
BibTeX
@misc{wu2025lag,
  author = {Jian Wu and Jiayu Zhang and Dongyuan Li and Linyi Yang and Aoxiao Zhong and Renhe Jiang and Qingsong Wen and Yue Zhang},
  title = {{LAG: LLM agents for Leaderboard Auto Generation on Demanding}},
  year = {2025},
  eprint = {2502.18209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.18209},
}
Figure