Overview of the 2021 SDP 3C Citation Context Classification Shared Task

Suchetha N. Kunnath, David Pride, Drahomira Herrmannova, Petr KnothNAACL 2021

TL;DR

This paper summarizes the 2021 3C Citation Context Classification Shared Task, a competition focused on automatically understanding why a paper is cited and how influential that citation is. The task saw significant improvements over the previous year, with deep learning models, particularly transformer-based ones like SciBERT, leading the charge in achieving higher performance.

What Problem Does This Paper Address?

Traditional methods for evaluating research often rely on simple citation counts, which can be a blunt instrument. Citations, however, carry nuanced meanings: a paper might be cited to support a claim, contrast with a previous idea, or simply to acknowledge its existence. Understanding the purpose and influence of a citation from its surrounding text (the citation context) offers a richer way to assess research impact. However, comparing different approaches in this field has been difficult due to a lack of standardized tasks and datasets. The 3C shared task aims to address this by providing a common platform and benchmark for developing and evaluating citation context classification methods.

Methods

The shared task featured two subtasks:

  • Subtask A (Purpose Classification): A multi-class classification problem where citations are categorized into one of six classes: BACKGROUND, COMPARES_CONTRASTS, EXTENSION, FUTURE, MOTIVATION, and USES.
  • Subtask B (Influence Classification): A binary classification problem to determine if a citation is INCIDENTAL (less important) or INFLUENTIAL.

Participants were encouraged to develop methods for these tasks, with submissions evaluated using the macro F1-score. The paper highlights that while previous years saw simpler methods like TF-IDF and Random Forests, this year's competition saw a strong shift towards deep learning, especially transformer-based models like SciBERT, which were particularly effective for Subtask A.

Datasets

The shared task utilized a portion of the Academic Citation Typing (ACT) dataset. This dataset was previously used in the 2020 iteration of the task. A key enhancement for the 2021 task was the provision of full-text documents for all papers in the training and test sets, allowing participants to extract richer contextual information beyond just the citing sentence. The dataset is described as having a "highly imbalanced nature" for the purpose classification task, making it more challenging than the influence task.

Key Insights and Ideas

  • Deep Learning Dominance: The 2021 task clearly demonstrated the superiority of deep learning models, particularly transformer-based architectures like SciBERT, over traditional machine learning approaches for citation context classification.
  • Full Text Matters: Providing full-text documents significantly aided participants, allowing for more sophisticated feature extraction and context understanding.
  • Challenges Remain: Despite advancements, the multi-domain nature of the ACT dataset and the prevalence of multiple citations within a single citing sentence continue to pose significant challenges for achieving even higher performance.
  • Public vs. Private Score Discrepancy: Figure 2 suggests that for Subtask A (purpose classification), there was a notable deviation between public and private leaderboard scores, indicating potential overfitting on the public test set for some teams. Subtask B showed less deviation, suggesting more robust models.

Key Results

  • Winning Scores: The highest reported macro F1-scores were 0.26973 for Subtask A (purpose) and 0.60025 for Subtask B (influence).
  • Participant Numbers: 22 teams participated in Subtask A, and 19 in Subtask B.
  • Performance Improvement: Compared to the previous year (2020), there was a considerable improvement in performance, with an 8.03% increase in the average macro F1-score for Subtask B. Table 6 provides a statistical summary of scores, showing higher maximum and mean scores for both tasks in 2021 compared to 2020.

Limitations and Future Work

  • Dataset Challenges: The authors acknowledge that the multi-domain nature of the ACT dataset and the presence of multiple citations within a single sentence are significant hurdles that need further attention to develop more robust systems.
  • Limited Paper Submissions: While many teams participated in the Kaggle competition, only a few submitted system description papers and source code, limiting the ability to fully analyze the methods used by all top performers. The paper notes that the top team for the influence task did not submit their solution.
  • Future Directions: The paper implicitly suggests that future work should focus on better handling of multi-domain text and complex citation structures. Further research into leveraging external datasets and advanced deep learning architectures is also a clear path forward.

Summary updated

BibTeX
@inproceedings{kunnath2021overview,
  author = {Suchetha N. Kunnath and David Pride and Drahomira Herrmannova and Petr Knoth},
  title = {{Overview of the 2021 SDP 3C Citation Context Classification Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure