Automating Citation Placement with Natural Language Processing and Transformers

Davide Buscaldi, Danilo Dessì, Enrico Motta, Marco Murgia, Francesco Osborne, Diego Reforgiato RecuperoSemTech4STLD@ESWC 2024 · OpenReview

In scientific writing, the decision of what to cite is a matter of domain expertise, but determining where to place a citation within a sentence is a technical skill governed by linguistic norms. A recent study by Buscaldi et al., presented at SemTech4STLD ’24, investigates the automation of this process using Transformer models and rule-based NLP heuristics.

The Two-Fold Challenge of Citation Placement

The authors frame the problem as two distinct natural language processing (NLP) tasks:

  1. Citation Required Task: A binary classification problem to determine if a given sentence requires a reference.
  2. Citation Placement Task: A granular sequence labeling problem to identify the specific token after which a citation marker should appear.

Methodology: Transformers and Heuristics

The research evaluates two primary architectural approaches:

  • Generative (GPT-2): This approach treats citation placement as a text-generation task. By iteratively inserting a [MASK] token, the model predicts whether a citation placeholder is appropriate at that position.
  • Discriminative (BERT): This approach treats the problem as Named Entity Recognition (NER), where each token is classified as either REGULAR (no citation follows) or CITATION (a citation follows).

To refine these models, the authors introduced NLP Heuristics using the SpaCy library. These rule-based post-processing steps correct common model errors, such as moving citations from the middle of a noun phrase to the end or removing citations that erroneously appear after verbs or specific keywords like "Table" or "Figure."

Data and Evaluation

The models were trained and tested on several datasets, including arXiv-80 and a custom-cleaned dataset of 9,000 Computer Science papers called s2orc-9k. For evaluation, a "Gold Standard" dataset of 170 manually annotated sentences was used, achieving a high inter-annotator agreement of 79.7%.

Key Results and Discussion

The study revealed a significant performance gap between generative and discriminative models. While the NER-based BERT model achieved high precision, it suffered from very low recall, making it too conservative for practical use.

Performance metrics for the Citation Required (left) and Citation Placement (right) tasks without heuristics.

The generative model (GM-s2orc) proved more effective at capturing the flow of scientific prose. The application of NLP heuristics provided a substantial boost to performance across all models, particularly in improving F1-scores.

Performance metrics after applying NLP heuristics, showing significant improvements in F1-scores for both tasks.

Specifically, the fine-tuned generative model reached an F1-score of 0.814 for the "Citation Required" task and 0.500 for the "Citation Placement" task after heuristic refinement. The results suggest that while Transformers are powerful, they benefit significantly from linguistic rules that encode the "etiquette" of scientific writing.

Conclusion and Future Directions

This research demonstrates a promising path toward AI-driven writing assistants that understand the structural norms of research papers. Future work aims to integrate these placement models with Knowledge Graphs to suggest specific papers and to incorporate "citation intent" to further refine the structural placement of references.

Summary updated

BibTeX
@inproceedings{buscaldi2024automating,
  author = {Davide Buscaldi and Danilo Dessì and Enrico Motta and Marco Murgia and Francesco Osborne and Diego Reforgiato Recupero},
  title = {{Automating Citation Placement with Natural Language Processing and Transformers}},
  booktitle = {SemTech4STLD@ESWC},
  year = {2024},
  url = {https://openreview.net/forum?id=jAYlZnewSr},
}

Loading related content...

Figure