In scientific writing, the decision of what to cite is a matter of domain expertise, but determining where to place a citation within a sentence is a technical skill governed by linguistic norms. A recent study by Buscaldi et al., presented at SemTech4STLD ’24, investigates the automation of this process using Transformer models and rule-based NLP heuristics.
The Two-Fold Challenge of Citation Placement
The authors frame the problem as two distinct natural language processing (NLP) tasks:
- Citation Required Task: A binary classification problem to determine if a given sentence requires a reference.
- Citation Placement Task: A granular sequence labeling problem to identify the specific token after which a citation marker should appear.
Methodology: Transformers and Heuristics
The research evaluates two primary architectural approaches:
- Generative (GPT-2): This approach treats citation placement as a text-generation task. By iteratively inserting a
[MASK]token, the model predicts whether a citation placeholder is appropriate at that position. - Discriminative (BERT): This approach treats the problem as Named Entity Recognition (NER), where each token is classified as either
REGULAR(no citation follows) orCITATION(a citation follows).
To refine these models, the authors introduced NLP Heuristics using the SpaCy library. These rule-based post-processing steps correct common model errors, such as moving citations from the middle of a noun phrase to the end or removing citations that erroneously appear after verbs or specific keywords like "Table" or "Figure."
Data and Evaluation
The models were trained and tested on several datasets, including arXiv-80 and a custom-cleaned dataset of 9,000 Computer Science papers called s2orc-9k. For evaluation, a "Gold Standard" dataset of 170 manually annotated sentences was used, achieving a high inter-annotator agreement of 79.7%.
Key Results and Discussion
The study revealed a significant performance gap between generative and discriminative models. While the NER-based BERT model achieved high precision, it suffered from very low recall, making it too conservative for practical use.
The generative model (GM-s2orc) proved more effective at capturing the flow of scientific prose. The application of NLP heuristics provided a substantial boost to performance across all models, particularly in improving F1-scores.
Specifically, the fine-tuned generative model reached an F1-score of 0.814 for the "Citation Required" task and 0.500 for the "Citation Placement" task after heuristic refinement. The results suggest that while Transformers are powerful, they benefit significantly from linguistic rules that encode the "etiquette" of scientific writing.
Conclusion and Future Directions
This research demonstrates a promising path toward AI-driven writing assistants that understand the structural norms of research papers. Future work aims to integrate these placement models with Knowledge Graphs to suggest specific papers and to incorporate "citation intent" to further refine the structural placement of references.
Summary updated