Abstract
Leaderboards are crucial in the machine learning (ML) domain for benchmarking and tracking progress. However, creating leaderboards traditionally demands significant manual effort. In recent years, efforts have been made to automate leaderboard generation, but existing datasets for this purpose are limited by capturing only the best results from each paper and limited metadata. We present MetaLead, a fully human-annotated ML Leaderboard dataset that captures all experimental results for result transparency and contains extra metadata, such as the result experimental type: baseline, proposed method, or variation of proposed method for experiment-type guided comparisons, and explicitly separates train and test dataset for cross-domain assessment. This enriched structure makes MetaLead a powerful resource for more transparent and nuanced evaluations across ML research.
Introduction: Moving Beyond "SOTA-Only" Leaderboards
In the fast-paced world of Machine Learning (ML), leaderboards are the primary mechanism for tracking progress. However, current leaderboards often suffer from "performance myopia." By typically reporting only the single best result (State-of-the-Art or SOTA) from a paper, they inadvertently encourage publication bias and obscure the context of how a model performs relative to its baselines and variations.
Existing datasets used to train automated leaderboard extraction systems are often too simplistic. They frequently fail to distinguish between training and test sets—making it impossible to track out-of-distribution (OOD) performance—and they lack categorization for the role of an experiment (e.g., distinguishing a main contribution from a minor ablation study).
The MetaLead Dataset
To address these limitations, we introduce MetaLead, a dataset built through a rigorous two-step human annotation process. First, annotators identify the core tuple (task, training dataset, test dataset, metric, and score) for every result reported in a paper. Second, each result is categorized by experiment type: Baseline, Proposed Method, or Variation of Proposed Method (ablations).
MetaLead represents a significant "vertical" expansion of leaderboard data. While previous datasets like SciLead focused on horizontal coverage (more papers), MetaLead extracts 3,568 annotated tuples across 43 papers—a 12x increase in depth for the same set of publications. This density reveals the "negative results" and "failed variants" crucial for a holistic understanding of model robustness.
Experimental Evaluation
We evaluated several Large Language Models (LLMs)—including GPT-4, Gemini 2.5, and Llama 3.3—on their ability to automate this extraction process. We tested them in both Closed Domain (predefined entities) and Open Domain (raw text extraction) scenarios.
Key Findings
- The Tuple Bottleneck: While LLMs are proficient at identifying individual entities (e.g., a specific score), their performance drops significantly when linking all six elements (Task, Train Set, Test Set, Metric, Score, Type) into a single correct tuple. A single incorrect link invalidates the entire data point.
- Experiment Type Classification: LLMs excel at identifying Baselines (often ~90% accuracy) but struggle to distinguish between the Main Proposed Method and its Ablation Variations. This suggests that ambiguous phrasing in research papers poses a challenge for both human and machine understanding.
- Performance Leaders: Gemini-2.5-Pro and GPT-4.1 emerged as the strongest models. In closed-domain settings, Gemini-2.5-Pro achieved a tuple-match F1 score of 48.89. However, performance plummeted in the open-domain "cold-start" scenario, with GPT-4.1 falling to an F1 of 14.14.
Discussion and Future Work
MetaLead highlights that extracting structured data "in the wild" remains a significant challenge for current AI. Even the best-performing model could only fully reconstruct about half of the "gold standard" leaderboards.
Future work should focus on expanding the dataset scale beyond the initial 43 papers and incorporating multi-modal data. Currently, the extraction process ignores figures and graphs, which contain a substantial portion of ML results. Integrating Vision-Language Models (VLMs) and multi-agent frameworks could provide the next leap in automated leaderboard accuracy and transparency.
Summary updated