Context-Enhanced Citation Sentiment Detection

Awais Athar, Simone TeufelNAACL 2012

TL;DR

This paper tackles the challenge of accurately detecting sentiment in scientific citations, arguing that focusing solely on the citation sentence misses crucial context. They introduce a new, context-aware annotated corpus and demonstrate that incorporating surrounding sentences significantly improves sentiment detection, especially for nuanced criticism.

What Problem Does This Paper Address?

Scientific papers often use citations to refer to previous work. Understanding the sentiment expressed in these citations (e.g., positive, negative, neutral) is valuable for applications like improved bibliographic search, bibliometrics, and even scientific summarization. However, detecting this sentiment is tricky. Scientific writing prioritizes objectivity, and authors may subtly embed criticism within praise or hedge their negative remarks. This means that the sentiment of a citation might not be fully captured by the sentence in which it appears. Previous work has largely focused on analyzing only the citation sentence itself, potentially leading to misinterpretations, especially when criticism is expressed indirectly or across multiple sentences.

Methods

The authors propose a "context-enhanced" approach to citation sentiment detection. Their core contribution is a new annotated corpus where the sentiment is determined by considering the dominant sentiment within a defined window of sentences surrounding the citation. They then evaluate a state-of-the-art citation sentiment detection system using this new corpus. Specifically, they explore the impact of different context window sizes (varying the number of preceding and succeeding sentences) on the system's performance. They also compare their context-enhanced approach to a baseline that only considers the citation sentence.

Datasets

The authors created a new benchmark dataset by annotating a subset of 20 papers from the ACL Anthology Network (comprising approximately 20% of the original dataset, which contained 8736 citations). This new corpus features a four-class annotation scheme: "excluded" (sentences not directly related to the citation), "objective/neutral" (0), "positive" (p), and "negative" (n). They also employed a three-class scheme for comparison, merging the context into a single sentence and assigning the sentiment of the last mentioned sentiment.

Key Insights and Ideas

  • Context is King (for Sentiment): Ignoring the surrounding sentences around a citation leads to a significant loss of sentiment information, particularly for negative sentiment. This is crucial because many citations are neutral, making the explicit sentiment that is present more valuable.
  • "Sweetened Criticism" is Real: The paper highlights that authors may strategically soften criticism with praise, making it difficult to detect sentiment from the citation sentence alone.
  • Fixed Window Limitations: While context is beneficial, using fixed-length context windows can be challenging. The authors observed that increasing context window size sometimes led to a decrease in performance, possibly due to increased vocabulary size and reduced discriminating power of features.
  • New Corpus is Closer to Truth: The authors argue their context-enhanced corpus is a more accurate reflection of the true sentiment than annotations that only consider the citation sentence.

Key Results

  • Using the context-enhanced corpus, the proportion of subjective sentiment (positive and negative) more than doubled compared to a context-less approach.
  • For negative sentiment specifically, the proportion rose threefold when context was considered.
  • The paper reports results for a Support Vector Machine (SVM) classifier. For joint sentiment and context detection (four-class annotation), the F-macro scores ranged from 0.672 to 0.768, and F-micro scores from 0.777 to 0.932, depending on the window size.
  • When using the three-class annotation scheme (context merged into one sentence), the best F-micro score achieved was 0.871 with a specific window configuration, though this was still slightly lower than the context-less baseline.

Limitations and Future Work

  • Data Sparsity: The authors acknowledge that existing work on citation sentiment analysis often uses more data, and their results with the three-class scheme were not better than the context-less baseline, potentially due to data sparsity.
  • Fixed Window Issues: The performance degradation with larger fixed context windows suggests that more intelligent filtering of context sentences is needed.
  • Future Directions: The authors suggest improving detection algorithms by more intelligently filtering context sentences, potentially using coreference resolution techniques. They also propose using context features to first filter out citations mentioned only in passing, and then applying context-based sentiment classification to the remaining significant citations.

Summary updated

BibTeX
@inproceedings{athar2012context,
  author = {Awais Athar and Simone Teufel},
  title = {{Context-Enhanced Citation Sentiment Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2012},
}
Figure