Mapping Subsets of Scholarly Information

Paul Ginsparg, Paul Houle, Thorsten Joachims, Jae-Hoon SularXiv 2003 · DOI · OpenReview

Abstract

We illustrate the use of machine learning techniques to analyze, structure, maintain, and evolve a large online corpus of academic literature. An emerging field of research can be identified as part of an existing corpus, permitting the implementation of a more coherent community structure for its practitioners.

Introduction

By 2003, the arXiv repository had become a cornerstone of scientific communication, hosting over 250,000 papers. However, the static nature of its categorization system struggled to keep pace with the fluid evolution of scientific disciplines. Specifically, the intersection of physics and biology was seeing a surge in activity, yet these papers were scattered across disparate categories like "Statistical Mechanics" or "Disordered Systems." This fragmentation created manual overhead for maintainers and hindered community coherence for researchers. We propose a machine learning approach to detect these emerging clusters and provide empirical evidence for formalizing new categories.

Methodology

We framed the identification of research subsets as a supervised text classification problem using Support Vector Machines (SVMs).

Feature Representation and Weighting

Documents were represented using a "bag-of-words" model, mapping titles and abstracts to high-dimensional vectors. To refine this representation, we applied TF×IDF (Term Frequency-Inverse Document Frequency) weighting to highlight significant terms and Porter stemming to group word variants. To address polysemy—where words like "evolution" have different meanings in physics versus biology—we expanded the vocabulary to include frequent two-word phrases with unique semantic content.

Calibration and Refinement

Because SVMs can be biased toward the majority class in small categories, we employed Platt scaling to convert raw decision boundary distances into calibrated probabilities. Additionally, we used the SVM's dual formulation to identify and manually inspect outlier documents, ensuring a high-quality training set.

Results

Our analysis focused on the "Quantitative Biology" (q-bio) subset. The SVM classifier successfully identified 1,649 q-bio documents that were previously "hidden" within other physics categories between 1992 and 2003.

Figure 6: SVM weights for the Quantitative Biology classifier, showing positive associations with biological terms and negative associations with unrelated physics terms.

As shown in Figure 6, the classifier learned to strongly associate terms like "protein," "dna," and "biological" with the new category, while assigning negative weights to terms like "quantum" or "equation" that are more characteristic of general physics.

Benchmarking and Scale

The system's performance was evaluated across the entire arXiv corpus. As illustrated in Figure 3, the classifier's effectiveness is closely tied to the volume of available training data.

Figure 3: Recall and precision plotted against the number of documents in a category, demonstrating higher performance for larger, more established fields.

Both recall and precision generally improve as the number of documents in a category increases. The classifier was remarkably successful for large categories (>10,000 documents), providing the empirical basis for the official launch of the q-bio archive in September 2003. Activity levels immediately following the launch matched the SVM's predictions almost exactly.

Discussion and Conclusion

While SVMs proved highly effective for large categories, performance degraded for very small or topically diffuse subject areas (<100 documents). Future work may explore Latent Semantic Analysis (LSA) to improve synonym handling or generative models like Latent Dirichlet Allocation (LDA), though computational efficiency remains a primary concern for real-time, large-scale deployment. Ultimately, this work demonstrates that machine learning can serve as a tool for "social engineering" in science, accelerating the growth of emerging fields by providing them with a formal institutional home.

Summary updated

Abstract

We illustrate the use of machine learning techniques to analyze, structure, maintain, and evolve a large online corpus of academic literature. An emerging field of research can be identified as part of an existing corpus, permitting the implementation of a more coherent community structure for its practitioners.
BibTeX
@misc{ginsparg2003mapping,
  author = {Paul Ginsparg and Paul Houle and Thorsten Joachims and Jae-Hoon Sul},
  title = {{Mapping Subsets of Scholarly Information}},
  year = {2003},
  eprint = {cs/0312018},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cs/0312018},
}
Figure