AxCell: Automatic Extraction of Results from Machine Learning Papers

Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, Robert Stojnic

Abstract: Tracking progress in machine learning has become increasingly difficult with the recent explosion in the number of papers. In this paper, we present AxCell, an automatic machine learning pipeline for extracting results from papers. AxCell uses several novel components, including a table segmentation subtask, to learn relevant structural knowledge that aids extraction. When compared with existing methods, our approach significantly improves the state of the art for results extraction. We also release a structured, annotated dataset for training models for results extraction, and a dataset for evaluating the performance of models on this task. Lastly, we show the viability of our approach enables it to be used for semi-automated results extraction in production, suggesting our improvements make this task practically viable for the first time.

Introduction

The rapid growth of machine learning research has created a "deluge of papers," making it nearly impossible for researchers to manually track state-of-the-art (SOTA) progress across various benchmarks. While platforms like Papers With Code provide valuable leaderboards, they currently rely heavily on crowdsourcing, which is labor-intensive and often slow to update.

The primary technical challenge in automating this process is that research results are typically trapped in semi-structured tables. A single table often contains a mix of main results, ablation studies, and comparative results from previous literature. Existing extraction tools often fail because they attempt to parse PDFs, which strips away the underlying structural information necessary to distinguish between these different types of data.

The AxCell Pipeline

AxCell addresses these challenges by moving away from brittle PDF parsing and instead utilizing LaTeX source code. This approach preserves the structural intent of the table (e.g., \toprule or specific cell alignments). The pipeline consists of several key stages designed to transform raw paper source into structured result tuples (task, dataset, metric, and value).

The AXCELL pipeline overview, showing stages from table extraction to final result filtering.
  1. Table Type Classification: A ULMFiT-based classifier determines if a table is a "leaderboard" table (containing main results), an "ablation" table, or irrelevant metadata.
  2. Table Semantic Segmentation: This core innovation treats table parsing as a semantic segmentation task. Individual cells are classified into roles such as dataset name, metric name, paper model, or cited model.
  3. Cell Context Generation: To interpret numerical values, the system identifies "mentions" of cell content within the paper's text using a BM25 scoring algorithm.
  4. Generative Linking Model: A Naive Bayes generative model links extracted evidence to a predefined taxonomy of known leaderboards (e.g., mapping "SST-2" to the Stanford Sentiment Treebank).

Datasets for Result Extraction

To support this research and future work, the authors released several high-quality datasets:

  • ARXIVPAPERS: Over 100,000 unlabeled ML papers for language model pre-training.
  • SEGMENTEDTABLES: 1,994 tables with manually annotated semantic cell roles.
  • LINKEDRESULTS: 239 papers containing 1,591 verified result tuples linked to specific tables.
  • PWC LEADERBOARDS: A large-scale evaluation set derived from over 2,000 Papers With Code leaderboards.

Key Results and Performance

AxCell represents a significant leap forward in extraction accuracy. By specifically training a classifier to identify and filter out "ablation traps"—lower-performing results from internal model variations—the system maintains high leaderboard integrity.

Performance comparison between AXCELL and the previous state-of-the-art (TDMS-IE).

On the NLP-TDMS dataset, AxCell achieved a micro F1 score of 25.8 for extracting (task, dataset, metric, score) tuples, a nearly 3.5x improvement over the previous state-of-the-art (TDMS-IE) which scored 7.5. The system is particularly robust at identifying entities, achieving over 70% precision in associating the correct "Task" and "Dataset" with a specific numerical result.

Conclusion

AxCell demonstrates that automated result extraction is becoming practically viable for production environments, particularly in "human-in-the-loop" settings. While the current system relies on LaTeX source code (available for approximately 90% of arXiv papers), it sets a new benchmark for structural table understanding. Future work aims to extend these capabilities to PDF parsing and develop methods for discovering new tasks and datasets that are not yet part of a predefined taxonomy.

Summary updated

Abstract

Tracking progress in machine learning has become increasingly difficult with the recent explosion in the number of papers. In this paper, we present AxCell, an automatic machine learning pipeline for extracting results from papers. AxCell uses several novel components, including a table segmentation subtask, to learn relevant structural knowledge that aids extraction. When compared with existing methods, our approach significantly improves the state of the art for results extraction. We also release a structured, annotated dataset for training models for results extraction, and a dataset for evaluating the performance of models on this task. Lastly, we show the viability of our approach enables it to be used for semi-automated results extraction in production, suggesting our improvements make this task practically viable for the first time. Code is available on GitHub.
BibTeX
@inproceedings{kardas2020axcell,
  author = {Marcin Kardas and Piotr Czapla and Pontus Stenetorp and Sebastian Ruder and Sebastian Riedel and Ross Taylor and Robert Stojnic},
  title = {{AxCell: Automatic Extraction of Results from Machine Learning Papers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=lq3n4v7X96},
}
Figure