Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction

Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Debasis Ganguly

Abstract

The rapid growth of scientific literature makes tracking the state-of-the-art (SOTA) for specific research tasks an increasingly difficult manual task. We present TDMS-IE, a framework designed to automatically construct scientific leaderboards by extracting Task, Dataset, Metric, and Score (TDMS) tuples from NLP research papers. By reframing information extraction as a Natural Language Inference (NLI) problem, our approach achieves significant improvements in accuracy and demonstrates strong generalization to unseen tasks.

Introduction: The Challenge of Tracking SOTA

AI research is expanding at an unprecedented rate, with thousands of papers published monthly. While platforms like Papers with Code provide invaluable community-curated leaderboards, the manual effort required to keep them updated is unsustainable. Scientific papers are typically unstructured PDFs where terminology varies widely—for instance, the same metric might be referred to as "F1," "F-score," or "F-measure."

Figure 1: The TDMS-IE framework extracts structured leaderboard annotations (Task, Dataset, Metric, Score) from unstructured scientific text and tables.

The goal of this work is to automate the extraction of these critical results to create dynamic, automated leaderboards without human intervention.

Methods: Information Extraction as Inference

The TDMS-IE system employs a BERT-based architecture divided into two primary classification components:

1. TDM Identification (DocTAET-TDM)

The core innovation of this paper is treating Information Extraction (IE) as a Natural Language Inference (NLI) task. Instead of traditional sequence tagging, the model is presented with a "premise" (the paper's title, abstract, and experimental setup) and a "hypothesis" (a specific Task-Dataset-Metric triple from a predefined taxonomy). The model determines if the paper entails the triple. This semantic approach allows the model to bridge the gap between different terminologies used by various authors.

2. Score Extraction (SC-DM)

Once the TDM triple is identified, the system analyzes the paper's tables to find the corresponding numeric results. It specifically targets boldfaced numbers, which typically denote the best performance. Using "Score Context"—the surrounding table captions and column headers—the model links the correct numeric score to the identified TDM triple.

Figure 3: The training and inference stages of the TDMS-IE framework, highlighting the NLI-based TDM identification and the score context mapping.

Experimental Datasets

The authors evaluated TDMS-IE using two newly developed datasets:

  • NLP-TDMS: A high-quality dataset derived from the NLP-progress repository, containing 332 papers and 848 leaderboard annotations across 168 distinct leaderboards.
  • ARC-PDN: A large-scale dataset of 3,855 papers from the ACL Anthology (2010–2015), used to test the model's scalability in a realistic, diverse research library.

Key Insights and Results

The shift to an NLI-based framework yielded several critical findings:

  • Superior Accuracy: TDMS-IE significantly outperformed traditional baselines. For TDM extraction, it achieved a Macro F1 of 65.3, more than doubling the performance of simple string-matching methods (31.0).
  • Context Matters: Ablation studies revealed that including "Experimental Setup" and "Table Info" (captions and headers) in the document representation was vital, nearly sextupling the Macro F1 score compared to using only the Title and Abstract.
  • Zero-Shot Generalization: Because the model learns semantic relationships rather than just label patterns, it can identify TDM triples it has never encountered during training, achieving a 41.6 Macro F1 in zero-shot scenarios.
  • Scalability: On the large ARC-PDN dataset, the model maintained a Precision@1 of 0.70, demonstrating its robustness against the noise of large-scale digital libraries.

Limitations and Future Directions

Despite its success, several challenges remain. Extracting the exact numeric score is difficult (F1 of 9.9) due to the inherent complexity of parsing PDF table structures and correctly associating column headers with values. Furthermore, while the system identifies the best score, it does not yet consistently link that score to the specific method or model name proposed in the paper. Future work will focus on improving table structure decomposition and expanding the TDM taxonomy to other scientific domains beyond NLP.

Summary updated

Abstract

While the fast-paced inception of novel tasks and new datasets helps foster active research in a community towards interesting directions, keeping track of the abundance of research activity in different areas on different datasets is likely to become increasingly difficult. The community could greatly benefit from an automatic system able to summarize scientific results, e.g., in the form of a leaderboard. In this paper we build two datasets and develop a framework (TDMS-IE) aimed at automatically extracting task, dataset, metric and score from NLP papers, towards the automatic construction of leaderboards. Experiments show that our model outperforms several baselines by a large margin. Our model is a first step towards automatic leaderboard construction, e.g., in the NLP domain.
BibTeX
@inproceedings{hou2019identification,
  author = {Yufang Hou and Charles Jochim and Martin Gleize and Francesca Bonin and Debasis Ganguly},
  title = {{Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=tXgLZEeAtEU},
}
Figure