Abstract
The rapid growth of scientific literature makes tracking the state-of-the-art (SOTA) for specific research tasks an increasingly difficult manual task. We present TDMS-IE, a framework designed to automatically construct scientific leaderboards by extracting Task, Dataset, Metric, and Score (TDMS) tuples from NLP research papers. By reframing information extraction as a Natural Language Inference (NLI) problem, our approach achieves significant improvements in accuracy and demonstrates strong generalization to unseen tasks.
Introduction: The Challenge of Tracking SOTA
AI research is expanding at an unprecedented rate, with thousands of papers published monthly. While platforms like Papers with Code provide invaluable community-curated leaderboards, the manual effort required to keep them updated is unsustainable. Scientific papers are typically unstructured PDFs where terminology varies widely—for instance, the same metric might be referred to as "F1," "F-score," or "F-measure."
The goal of this work is to automate the extraction of these critical results to create dynamic, automated leaderboards without human intervention.
Methods: Information Extraction as Inference
The TDMS-IE system employs a BERT-based architecture divided into two primary classification components:
1. TDM Identification (DocTAET-TDM)
The core innovation of this paper is treating Information Extraction (IE) as a Natural Language Inference (NLI) task. Instead of traditional sequence tagging, the model is presented with a "premise" (the paper's title, abstract, and experimental setup) and a "hypothesis" (a specific Task-Dataset-Metric triple from a predefined taxonomy). The model determines if the paper entails the triple. This semantic approach allows the model to bridge the gap between different terminologies used by various authors.
2. Score Extraction (SC-DM)
Once the TDM triple is identified, the system analyzes the paper's tables to find the corresponding numeric results. It specifically targets boldfaced numbers, which typically denote the best performance. Using "Score Context"—the surrounding table captions and column headers—the model links the correct numeric score to the identified TDM triple.
Experimental Datasets
The authors evaluated TDMS-IE using two newly developed datasets:
- NLP-TDMS: A high-quality dataset derived from the NLP-progress repository, containing 332 papers and 848 leaderboard annotations across 168 distinct leaderboards.
- ARC-PDN: A large-scale dataset of 3,855 papers from the ACL Anthology (2010–2015), used to test the model's scalability in a realistic, diverse research library.
Key Insights and Results
The shift to an NLI-based framework yielded several critical findings:
- Superior Accuracy: TDMS-IE significantly outperformed traditional baselines. For TDM extraction, it achieved a Macro F1 of 65.3, more than doubling the performance of simple string-matching methods (31.0).
- Context Matters: Ablation studies revealed that including "Experimental Setup" and "Table Info" (captions and headers) in the document representation was vital, nearly sextupling the Macro F1 score compared to using only the Title and Abstract.
- Zero-Shot Generalization: Because the model learns semantic relationships rather than just label patterns, it can identify TDM triples it has never encountered during training, achieving a 41.6 Macro F1 in zero-shot scenarios.
- Scalability: On the large ARC-PDN dataset, the model maintained a Precision@1 of 0.70, demonstrating its robustness against the noise of large-scale digital libraries.
Limitations and Future Directions
Despite its success, several challenges remain. Extracting the exact numeric score is difficult (F1 of 9.9) due to the inherent complexity of parsing PDF table structures and correctly associating column headers with values. Furthermore, while the system identifies the best score, it does not yet consistently link that score to the specific method or model name proposed in the paper. Future work will focus on improving table structure decomposition and expanding the TDM taxonomy to other scientific domains beyond NLP.
Summary updated