TL;DR
With thousands of academic papers published daily, keeping track of the state-of-the-art has become an impossible task for human researchers. LEAGUE (Leaderboard Auto Generation) is a novel framework that leverages Large Language Models (LLMs) to automatically crawl, extract, and dynamically generate up-to-date research leaderboards. By capturing not just scores but also experimental settings, LEAGUE ensures fair comparisons at a fraction of the time and cost of manual curation.
What Problem Does This Paper Address?
In fast-moving fields like Artificial Intelligence, staying abreast of the state-of-the-art (SOTA) is overwhelming. While platforms like Papers with Code are incredibly valuable, they rely heavily on manual submissions and curation. As a result, a massive gap has formed between the surge of paper submissions and the updates of corresponding leaderboards. For instance, the authors note that the popular HotpotQA multi-hop question-answering leaderboard has remained stagnant since 2023, despite dozens of newer methods being published.
Previous attempts to automate this process focused purely on Scientific Named Entity Recognition (NER)—extracting isolated tasks, datasets, and models. However, these approaches yield static snapshots and fail to address three critical challenges:
- Paper Coverage: Finding all relevant papers on a rapidly evolving topic.
- Fair Comparison: Ensuring models are compared under standardized evaluation settings (accounting for variations in model size, training data volume, and hyperparameters).
- Timeliness: Keeping the leaderboard continuously updated.
Methods
LEAGUE introduces a systematic, four-stage pipeline designed to build and maintain leaderboards end-to-end:
- Paper Collection and Split: The system automatically crawls papers from arXiv and top-tier conferences based on a target topic. To prevent token bloat and stay within LLM context windows, it strips out unproductive LaTeX code and isolates the "Experiment" sections.
- Table Extraction and Classification: Using few-shot Chain-of-Thought (CoT) prompting, an LLM categorizes tables into "Main Results," "Ablation Study," or "Others," filtering out everything except the core performance tables.
- Table Unpacking and Integration: The framework extracts data into a structured quintuple: [Paper Title, Dataset, Metrics, Experiment Settings, Experiment Results]. Extracting experimental settings (like model size or data subsets) is the key innovation that enables fair baseline alignment.
- Leaderboard Generation and Evaluation: Candidate leaderboards are refined, deduplicated, and ranked by a third-party LLM. Crucially, the framework runs in an iterative loop (typically 5 iterations), using an LLM-as-a-judge to continuously score and optimize the leaderboard's quality.
Compared to prior work, LEAGUE stands out by being fully dynamic, multi-document capable, and aware of experimental settings:
Datasets
The authors evaluated LEAGUE by harvesting papers from open-access platforms and top-tier AI venues (including ACL, EMNLP, NeurIPS, ICML, ICLR, and NAACL). They focused on generating leaderboards for specific benchmarks, including:
- HotpotQA: For multi-hop question answering.
- LA Dataset: For semi-supervised medical image segmentation.
- LIVE Dataset: For image quality assessment.
To validate the intermediate extraction steps (Table Classification and Table NER), the authors also manually annotated a validation dataset consisting of 354 tables from 72 papers (containing 172 methods, 31 datasets, 49 settings, and 84 metrics).
Key Insights and Ideas
- Context Matters for Fairness: Simply extracting a number from a table isn't enough. LEAGUE demonstrates that capturing the experimental settings column is vital; otherwise, a small model might be unfairly penalized against a massive model, or a zero-shot method compared directly against a fully fine-tuned one.
- LLMs are Reliable Judges of Structure: The paper reveals a strong positive correlation (up to 0.76 Pearson correlation using
o1-preview) between human expert evaluations and LLM-generated quality scores, proving that automated evaluation can serve as a reliable proxy for human judgment. - The Power of Iteration: The authors found that looping the generation process up to 5 times drastically improved the logical structure and coverage of the leaderboards, as the model learned to refine missing items and discard redundant metrics over time.
Key Results
LEAGUE achieves performance remarkably close to manual human curation while offering a massive leap in efficiency:
- Speedup: Constructing a comprehensive 20-item leaderboard takes a human expert over 18 minutes. LEAGUE (using GPT-4o) accomplishes this in under 2 minutes—a 5x to 10x speedup.
- Quality: For a 20-item leaderboard, LEAGUE achieved a 67.58% recall and 70.33% precision in topic-related quality. Its content quality scores closely mirrored human writing across metrics like coverage, structure, and timeliness.
- Cost Efficiency: Generating a complete, 20-entry leaderboard costs less than $14 in API tokens, making it highly practical for continuous deployment.
Limitations and Future Work
Despite its success, the authors note a few limitations:
- Retrieval Dependence: LEAGUE is inherently bound by the quality of its initial paper crawler. If the retrieval model misses a key paper during Stage 1, it will never make it to the leaderboard.
- Metric Fragmentation: Different papers often use slightly different metrics or abbreviations to evaluate the same task. While LEAGUE attempts to align these, extreme variations or sparse data (where >60% of papers omit a metric) force the system to discard that metric entirely to maintain leaderboard clarity.
Future work will focus on developing more sophisticated retrieval models to maximize paper coverage and improving cross-paper metric normalization.
Summary updated