Abstract
Tracking progress in machine learning has become increasingly difficult with the recent explosion in the number of papers. In this paper, we present AxCell, an automatic machine learning pipeline for extracting results from papers. AxCell uses several novel components, including a table segmentation subtask, to learn relevant structural knowledge that aids extraction. When compared with existing methods, our approach significantly improves the state of the art for results extraction. We also release a structured, annotated dataset for training models for results extraction, and a dataset for evaluating the performance of models on this task. Lastly, we show the viability of our approach enables it to be used for semi-automated results extraction in production, suggesting our improvements make this task practically viable for the first time.
Introduction
The machine learning field is currently experiencing a "paper explosion," with tens of thousands of articles published annually. Tracking which model actually holds the "State of the Art" (SOTA) on a specific leaderboard has become a Herculean task for humans. While platforms like Papers With Code use crowdsourcing to maintain leaderboards, manual entry is slow and prone to errors. Previous attempts at automation were often brittle because they relied on extracting data from PDF files, which lose semantic formatting. AxCell is a new open-source pipeline that automatically extracts performance results (task, dataset, metric, and value) from machine learning papers by leveraging their LaTeX source code to make result extraction robust enough for production-level use.
Methods
AxCell treats result extraction as a multi-stage pipeline rather than a single "black box" model. By breaking the problem into specialized subtasks, it achieves a massive performance jump over previous methods.
- LaTeX Extraction: Instead of messy PDFs, AxCell uses LaTeX source code from arXiv, providing clean structural information about tables and text.
- Table Type Classification: A ULMFiT-based classifier identifies if a table is a leaderboard (main results), an ablation study, or irrelevant (e.g., hyperparameters).
- Table Semantic Segmentation: This is a core innovation. The model classifies individual table cells into categories: dataset, metric, paper model, cited model, or other.
- Context Generation: The system gathers evidence for a result from a "hierarchy of context," ranging from the specific table cell and its row/column headers to the table caption, text mentions, and the abstract.
- Linking & Filtering: Using Bayesian inference, the system matches extracted "evidence" against a known taxonomy of leaderboards. It then filters out results for older, cited models to keep only the new results introduced by the paper.
Datasets
The researchers utilized and released several key datasets to support this task:
- ArxivPapers: A massive unlabelled dataset of 100,000+ ML papers used for language model pre-training.
- SegmentedTables: 1,400 tables from 354 articles with ground-truth cell-level annotations.
- LinkedResults: 236 papers annotated with 1,148 result tuples linked directly to their source tables.
- PWC Leaderboards: A dataset of over 2,000 leaderboards from Papers With Code used for end-to-end evaluation.
Key Results
AxCell demonstrates a significant improvement over existing baselines. On the NLP-TDMS (Exp) benchmark, AxCell achieved a micro F1 score of 25.8, nearly 3.5x better than the previous best of 7.5. The table segmentation model also performed remarkably well across different domains, achieving F1 scores between 77% and 92% for various tasks like NLI and Object Detection.
Discussion and Insights
Several key insights emerged from this work. First, LaTeX is the "Cheat Code": by using LaTeX source, the authors bypass the "PDF-to-text" nightmare. Over 90% of arXiv papers have source code available, making this a highly viable strategy. Second, Segmentation is the Bottleneck: correctly identifying the role of a table cell (e.g., "Is this a dataset name or a model name?") is the hardest part of the process.
While AxCell currently assumes a "closed world" where leaderboards are known in advance, its performance suggests it is robust enough for semi-automated "human-in-the-loop" settings. Future work involves training the model to work directly with PDF files without losing accuracy and expanding its ability to capture fine-grained data from ablation studies.
Summary updated