Abstract
The number of scientific articles is increasing tremendously across all domains to such an extent that it has become hard for researchers to remain up-to-date. Evidently, scientific language understanding systems and Information Extraction (IE) systems, with the advancement of Natural Language Processing (NLP) techniques, are benefiting the needs of users. Although the majority of the practices for building such systems are data-driven, advocating the idea of “The more, the better”. In this work, we revisit the paradigm - questioning what type of data: text (title, abstract) or citations, can have more impact on the performance of scientific language understanding systems.
Introduction
The scientific community is currently navigating a "data deluge." With millions of papers published annually, researchers can no longer keep up manually. This has led to a surge in Scientific Language Understanding (SLU) systems designed to assist with retrieval, recommendation, and summarization.
The prevailing wisdom in AI often suggests that "the more data, the better." However, scientific documents are uniquely complex, containing unstructured text, dense terminology, and rich metadata like citation links. This study tackles a fundamental question: Which features actually drive performance? Is it the textual content (titles and abstracts) or the structural position of a paper within a citation network? By benchmarking these features, we aim to identify the most efficient strategies for building high-performance models for scientific text mining.
Methodology
We benchmarked three categories of features using a variety of machine learning and deep learning architectures on the PubMed-Diabetes dataset (19,717 articles; 44,338 citation edges):
- Text-only (T): We compared traditional TF-IDF (using curated keywords,
tfidf-c, and automatically generated ones,tfidf-m) against state-of-the-art Pre-trained Language Models (PLMs) like BERT, BioBERT, PubMedBERT, and BioElectra. We also evaluated a BERT-CNN architecture to capture local spatial features. - Graph-only (G): We utilized DeepWALK (DW), which treats a citation network like a language by performing "random walks" to create embeddings based on structural neighborhoods.
- Hybrid (T+G): We explored combining text and graph features through simple concatenation and GraphSAGE (GS), a framework that uses iterative message passing to aggregate information from a paper’s neighbors.
Results and Analysis
The experimental results, summarized in the table below, provide a comprehensive view of how different data sources impact classification performance.
The "Abstract" Advantage
Unsurprisingly, abstracts consistently outperform titles. Models trained on abstracts generally see a 10-20% jump in F1-score compared to those trained only on titles. The top performer, PubMedCNN, achieved an F1-score of 0.87 when processing abstracts.
Citations as a "Semantic Shortcut"
A striking finding is that DeepWALK (graph-only) performs nearly as well as TF-IDF (text-only). DeepWALK achieved an F1-score of 0.65 without "reading" a single word of the paper, suggesting that a paper's citation context is a powerful proxy for its topic.
The Importance of Domain-Specific Vocabulary
PubMedBERT, trained from scratch on biomedical text, outperformed models like BioBERT. This highlights the necessity of "in-domain" vocabulary for identifying complex medical terms that general-purpose models often struggle to process.
The Hybrid Boost and the Noise of Message Passing
When working with limited text (titles only), adding citation information provided a significant lift. For instance, combining curated TF-IDF with GraphSAGE (tfidf-c+GS) boosted the title-only F1-score from 0.67 to 0.76. However, using GraphSAGE with raw TF-IDF features sometimes decreased performance compared to text-only models, likely because the iterative "averaging" of features from neighbors can wash out specific, high-quality information in the original text.
Conclusion
This study demonstrates that while abstracts remain the most informative single source of text, citation graphs provide a surprisingly robust signal for document classification. For tasks where full abstracts may be unavailable, citation-based features offer a powerful alternative or supplement. Future work should explore whether including full-text sections provides significant breakthroughs and how dynamic, evolving citation networks can be used to detect emerging research trends.
Summary updated