unarXive 2022: All arXiv Publications Pre-Processed for NLP, Including Structured Full-Text and Citation Network

Tarek Saier, Johan Krause, Michael Färber

Introduction

The rapid growth of scientific literature presents a significant challenge for knowledge discovery and natural language processing (NLP). While the arXiv repository serves as a primary hub for STEM research, its contents are predominantly distributed as PDFs—a format optimized for human reading but notoriously difficult for machine parsing. Existing datasets often strip away crucial document structure, mangle mathematical notation, or fail to provide a robust citation network.

To address these limitations, the unarXive 2022 dataset provides a structured, large-scale resource featuring 1.9 million publications spanning 32 years. By preserving document hierarchy, retaining LaTeX mathematical notation, and linking citations to the OpenAlex global database, unarXive 2022 offers a "goldilocks" solution for scholarly NLP: large-scale, structurally rich, and mathematically literate.

Methodology: From LaTeX to Structured JSON

The authors developed a sophisticated pipeline to transform raw LaTeX source files into a clean, machine-readable format. This process involves several key stages:

  1. Parsing and Normalization: Using latexpand, complex multi-file LaTeX projects are flattened into single documents. These are then converted into XML using Tralics, a LaTeX-to-XML translator.
  2. Structural Preservation: The final output is a nested JSON format that maintains the document's logical hierarchy, such as the relationship between sections, subsections, and paragraphs. Unlike "bag-of-words" approaches, this preserves the context of specific claims (e.g., whether a statement appears in an "Introduction" or a "Proof").
  3. Feature Enrichment: The pipeline explicitly identifies and marks citation markers, tables, figures, and mathematical formulas. Crucially, math is kept in its original LaTeX form, enabling research into mathematical NLP.
  4. Reference Linking: To construct a high-quality citation network, the authors utilized GROBID to parse bibliography strings. These were then matched against OpenAlex, the open-source successor to the Microsoft Academic Graph. This integration allows researchers to link unarXive text with external metadata, such as author affiliations and citation impact.

Dataset Characteristics and Benchmarks

The core unarXive 2022 dataset comprises 1.9 million papers, 182 million paragraphs, and 133 million in-text citation markers. Beyond the primary dataset, the authors released two specialized benchmarks for common NLP tasks:

  • Citation Recommendation Task: A collection of 2.5 million items consisting of a paragraph and a citation marker, designed to train models that suggest relevant literature based on local context.
  • IMRaD Classification Task: A dataset of 530,000 paragraphs labeled by section type (Introduction, Methods, Results, and Discussion + Related Work). This supports models in understanding the functional roles of different text segments.

Key Insights and Results

The integration with OpenAlex proved highly effective, yielding a 44.4% citation completeness rate—the highest among comparable arXiv-based datasets. This makes unarXive 2022 a premier resource for bibliometric analysis and citation analysis.

Analysis of the dataset reveals shifting trends in the scientific landscape. While the corpus is currently dominated by Physics (57%), Mathematics (20%), and Computer Science (17%), Computer Science submissions are projected to surpass Physics by 2023. Furthermore, the "density" of citations (references per paragraph) is increasing rapidly in Computer Science and Physics, while remaining relatively stable in Mathematics.

Limitations and Future Directions

Despite its scale, unarXive 2022 has limitations. The dataset is heavily skewed toward the physical sciences and computer science, lacking the representation of biomedical or social sciences found in datasets like PMC-OAS. Additionally, because LaTeX is a complex, Turing-complete language, no parser is perfect; some extraction errors persist in documents with highly non-standard formatting.

Finally, licensing remains a constraint. While a subset of 165,000 papers is fully open access, the complete dataset is available under a restricted access policy on Zenodo to comply with arXiv's varied licensing terms. Future work will focus on incremental updates and deeper analysis of the structured mathematical notation within the corpus.

Summary updated

Abstract

Large-scale data sets on scholarly publications are the basis for a variety of bibliometric analyses and natural language processing (NLP) applications. Especially data sets derived from publication's full-text have recently gained attention. While several such data sets already exist, we see key shortcomings in terms of their domain and time coverage, citation network completeness, and representation of full-text content. To address these points, we propose a new version of the data set unarXive. We base our data processing pipeline and output format on two existing data sets, and improve on each of them. Our resulting data set comprises 1.9 $\mathrm{M}$ publications spanning multiple disciplines and 32 years. It furthermore has a more complete citation network than its predecessors and retains a richer representation of document structure as well as non-textual publication content such as mathematical notation. In addition to the data set, we provide ready-to-use training/test data for citation recommendation and IMRaD classification. All data and source code is publicly available at https://github.com/IlIDepence/unarXive.
BibTeX
@inproceedings{saier2023unarxive,
  author = {Tarek Saier and Johan Krause and Michael Färber},
  title = {{unarXive 2022: All arXiv Publications Pre-Processed for NLP, Including Structured Full-Text and Citation Network}},
  booktitle = {JCDL},
  year = {2023},
  url = {https://openreview.net/forum?id=fdiw6SGXCe},
}
Figure