Purpose and Polarity of Citation: Towards NLP-based Bibliometrics

Amjad Abu-Jbara, Jefferson Ezra, Dragomir R. RadevNAACL 2013 · OpenReview

TL;DR

This paper introduces a novel approach to enrich bibliometric analysis by using Natural Language Processing (NLP) to understand the why and how behind citations. By analyzing the text surrounding a citation, the authors can determine the purpose and sentiment of the citation, moving beyond simple citation counts to a more nuanced understanding of research impact.

What Problem Does This Paper Address?

Traditional bibliometric measures, like the H-index or Impact Factor, rely solely on the quantity of citations to gauge a paper's impact. However, this approach has a significant limitation: it treats all citations equally, regardless of whether they are positive endorsements, critical evaluations, or mere mentions. This can lead to an inaccurate assessment of a paper's true influence, as even fraudulent or controversial work can garner a high number of citations. The authors argue that a more qualitative assessment is needed, and this paper aims to bridge that gap by using NLP to analyze the "citation context" – the text surrounding a citation – to understand the author's intent (purpose) and sentiment (polarity) behind the citation.

Methods

The paper proposes a three-stage NLP pipeline:

  1. Citation Context Identification: This involves identifying the relevant sentences around an explicit citation that discuss the cited work. The authors frame this as a sequence labeling problem, using Conditional Random Fields (CRFs) to label sentences as either "INCLUDED" or "EXCLUDED" from the citation context. They found a window of four sentences (the citing sentence, one before, and two after) to be optimal.
  2. Citation Purpose Classification: Once the context is identified, the authors classify the reason for the citation. They adopt a taxonomy with six categories: Criticizing, Comparison, Use, Substantiating, Basis, and Neutral (Other). This is treated as a supervised classification task, using lexical and structural features extracted from the citation context.
  3. Citation Polarity Classification: This task determines the sentiment of the citation towards the cited work. The possible polarities are positive, negative, or neutral. Similar to purpose classification, this is a supervised task using the same features. They found that classifying into "Polarized (Subjective)" vs. "Neutral (Objective)" first, and then distinguishing between positive and negative, yielded more intuitive results than a direct three-way classification.

A crucial preprocessing step involves Reference Tagging and Grouping to handle the complex formatting of references within scientific text, preventing parsing errors and improving feature extraction.

Datasets

  • ACL Anthology Network (AAN) Corpus: This is the primary dataset used for training and evaluation. It contains over 19,000 NLP papers with a curated citation network, full text, and citing sentences.
  • Training/Testing Dataset: Derived from 30 highly cited papers from AAN, resulting in approximately 3,500 citations and 14,000 sentences.
  • Development Dataset: A smaller dataset of 300 citations from 5 AAN papers, used for determining optimal window sizes and feature sets.

Key Insights and Ideas

  • Beyond Citation Counts: The core idea is that the text of a citation provides crucial qualitative information about a paper's impact, which simple counts miss.
  • Context is King: Analyzing only the direct citing sentence is insufficient. Including surrounding sentences significantly improves the accuracy of sentiment and purpose classification, especially for subjective citations.
  • NLP for Bibliometrics: This work demonstrates the power of NLP techniques to move bibliometrics from purely quantitative to a more qualitative and nuanced field.
  • Subjectivity Matters: Negative and positive citations (subjective) are more informative than neutral ones, and capturing these nuances is key to understanding research impact.

Key Results

  • Citation Context Identification: The proposed CRF-based sequence labeling approach significantly outperformed baselines (including an SVM-based method) in identifying citation context, achieving an F1 score of 89.5%.
  • Citation Purpose Classification: SVM achieved an overall accuracy of 70.5% with a Macro-F measure of 58.0% for the six purpose categories.
  • Citation Polarity Classification: SVM achieved an overall accuracy of 81.4% (74.2% without context) and a Macro-F of 71.3% (62.1% without context). The results showed a substantial improvement when using citation context, particularly for subjective categories like negative sentiment.
  • Feature Importance: Lexical features (like closest verb/adjective/adverb, subjectivity cues) were generally found to be more important than structural features for both purpose and polarity classification.

Limitations and Future Work

  • Dataset Size: The authors note that accuracy increases with more training data, suggesting that larger datasets could lead to even better performance.
  • Future Applications: The authors envision using this research for:
  • Predicting future prominence of publications.
  • Studying the dynamics of research trends.
  • Developing more accurate bibliometric measures.
  • Applications in citation-based summarization and survey generation systems.
  • Temporal Analysis: The paper briefly touches on the temporal analysis of citation purpose and polarity, suggesting it's useful for understanding research dynamics and the emergence/de-emergence of techniques.

Summary updated

BibTeX
@inproceedings{abujbara2013purpose,
  author = {Amjad Abu-Jbara and Jefferson Ezra and Dragomir R. Radev},
  title = {{Purpose and Polarity of Citation: Towards NLP-based Bibliometrics}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
  url = {https://openreview.net/forum?id=WI3JoXOcFdH},
}
Figure