TL;DR
This paper tackles the challenge of automatically classifying the purpose and importance of citations within scientific papers. By fine-tuning the SciBERT language model on citation context, the authors achieved state-of-the-art results on a shared task, demonstrating the power of domain-specific pre-trained models for understanding scientific literature.
What Problem Does This Paper Address?
The explosion of scientific literature makes it increasingly difficult for researchers to navigate and understand the relationships between papers. Identifying why a paper cites another (e.g., for background, comparison, or to extend previous work) and how important that citation is (incidental vs. influential) is crucial for tasks like literature review, knowledge discovery, and impact assessment. While manual classification is possible, it's time-consuming and doesn't scale. This paper addresses the need for automated methods to classify citation contexts, specifically focusing on the purpose and influence of citations.
Methods
The core of the paper's approach is leveraging pre-trained language models, specifically SciBERT, which is BERT pre-trained on a large corpus of scientific documents. The authors experimented with several methods:
- Fine-tuning BERT-based models: They fine-tuned BERT, RoBERTa, and SciBERT on the citation context data. A simple linear layer was used to classify the sentence embeddings generated by these models.
- SciBERT with Bi-LSTMs: They also explored using a Bi-directional Long Short-Term Memory (Bi-LSTM) network on top of SciBERT embeddings for classification.
- Random Forest: As a baseline and to explore simpler models, they used a Random Forest classifier with SciBERT embeddings.
Crucially, they found that using only the "citation context" (the sentence containing the citation) as input yielded the best results, even outperforming models that also included the cited paper's title. They also employed a weighted loss function to handle the imbalanced nature of the dataset.
Datasets
The primary dataset used is from the 3C-Citation Context Classification shared task of the Scholarly Document Processing Workshop at NAACL 2021. This dataset contains nine fields, but the authors focused on the "citation context" and "citation class label" (for subtask A) and "citation influence label" (for subtask B). They randomly split the 3000 instances into training (2500) and validation (500) sets, ensuring similar class distributions. For final evaluation, they combined the training and validation data.
Key Insights and Ideas
- Domain-specific pre-training matters: SciBERT, pre-trained on scientific text, significantly outperformed general-purpose BERT and RoBERTa models for this scientific text classification task. This highlights the value of tailoring language models to specific domains.
- Citation context is king: Surprisingly, the authors found that the citation context sentence alone was the most informative feature. Including the cited paper's title, which might seem intuitively useful, actually led to a drop in performance. This suggests that the immediate surrounding text of a citation is highly indicative of its purpose and influence.
- Simplicity can be effective: A fine-tuned SciBERT model with a simple linear layer achieved top performance, demonstrating that complex architectures aren't always necessary when using powerful pre-trained embeddings.
Key Results
The paper reports strong performance on the shared task's private leaderboard:
- Subtask A (Citation Purpose): Their best system, SciBERT-uncased + Linear, achieved a Macro F1 score of 0.26973 and ranked first.
- Subtask B (Citation Influence): Their best system, SciBERT-uncased + Linear, achieved a Macro F1 score of 0.59071 and ranked second.
They also noted that while BERT-uncased + Linear performed slightly better on their validation set for Subtask A, SciBERT-uncased + Linear ultimately outperformed it on the actual test data.
Limitations and Future Work
- Dataset Size and Noise: The authors acknowledge that the training dataset is relatively small and contains noise (e.g., numbers, HTML elements). They suggest that data augmentation techniques could be explored to improve results.
- Feature Engineering: While they found citation context to be sufficient, they mention that a more detailed study of other features could potentially enhance performance.
- Model Variance: They noted that fine-tuning BERT-like models can exhibit high variance, hence they ran their experiments multiple times and reported mean F1 scores and standard deviations.
Summary updated