Annotating with Pros and Cons of Technologies in Computer Science Papers

Hono Shirai, Naoya Inoue, Jun Suzuki, Kentaro Inui2019 · DOI

TL;DR

This paper introduces the challenging task of automatically identifying the pros and cons of technologies discussed in computer science papers. They developed an annotation scheme, created a dataset, and built a baseline model, finding that while annotation is consistent, automatic extraction is difficult due to the need for domain-specific knowledge.

What Problem Does This Paper Address?

In the ever-growing landscape of scientific publications, researchers struggle to keep up with the latest advancements and gain a quick overview of existing technologies. Computer science papers often present a problem and then discuss solutions, including the advantages (pros) and disadvantages (cons) of previously proposed technologies. Manually sifting through this information is time-consuming. This paper aims to automate the extraction of these pros and cons, making it easier for researchers to quickly understand the strengths and weaknesses of different technological approaches.

Methods

The authors adapted the targeted sentiment analysis task, a well-established Natural Language Processing (NLP) technique, for their purpose. They defined two key annotation labels:

  • TERM: This label identifies noun phrases that represent a mechanism, function, or method used to solve a problem (e.g., "recursive neural network," "AdaRNN").
  • Sentiment: For each identified TERM, they assigned a sentiment attribute: Positive, Negative, or Neutral. This sentiment is judged based solely on the information within the sentence containing the TERM.

They conducted an annotation study using three fluent English speakers specializing in NLP, with two annotators per paper to assess agreement. For the automatic extraction, they formulated the problem as a BIO sequence tagging task, where each word in a sentence is tagged to indicate whether it's the beginning (B), inside (I), or outside (O) of a Positive (POS), Negative (NEG), or Neutral (NEU) technical term span. They employed a BiLSTM-CRF model, a standard for named entity recognition, using both word2vec and ELMo embeddings.

Datasets

The authors created a new dataset by annotating the introduction sections of 92 computational linguistics papers that contained the keyword "coreference resolution." These papers were published between 1999 and 2017. The introduction sections were chosen to reduce annotation costs, as pros and cons are often discussed there. The annotated dataset is made publicly available.

Key Insights and Ideas

  • Pros/Cons as Targeted Sentiment: Framing the extraction of pros and cons as a targeted sentiment analysis task is a viable approach, showing reasonable annotation consistency.
  • Domain-Specific Knowledge is Crucial: The most significant challenge identified is the requirement for domain-specific knowledge. Annotators often needed to understand the nuances of coreference resolution or other CS sub-fields to correctly label sentiment, which is difficult for automated systems.
  • Implicit Sentiment is Tricky: The model struggled with sentences where sentiment was implied rather than explicitly stated, leading to misclassifications.
  • TERM Identification Challenges: Even identifying the "TERM" itself can be difficult when the phrase doesn't explicitly contain keywords indicating a mechanism or method, or when coreference resolution is needed to understand what a pronoun refers to.

Key Results

  • Annotation Consistency: The annotation study showed that pros and cons can be consistently annotated by human annotators. The exact match of TERM spans between annotators was 24.0%, with a partial match of 38.2%. For sentiment, Fleiss's Kappa was 0.65, indicating substantial agreement.
  • Model Performance: The best performing model (CL+EL) achieved F1 scores around 54% in a 10-fold cross-validation setting. Performance dropped in the "NEWYEAR" configuration (trained on older papers, tested on newer ones), highlighting the challenge of adapting to new terminology and concepts.
  • Qualitative Findings: Table 2 provides examples of model predictions, illustrating cases where the model failed to identify a TERM (e.g., "concept maps" without explicit keywords) or misclassified sentiment due to implicit expressions or lack of domain knowledge.

Limitations and Future Work

Limitations:

  • The automatic extraction task is challenging due to the reliance on domain-specific knowledge and inference, which current models lack.
  • The model struggles with implicit sentiment and identifying TERMS that don't contain explicit keywords.
  • Performance degrades when dealing with new or unseen terminology (as seen in the NEWYEAR configuration).

Future Work:

  • Expand annotation to other domains like computer vision.
  • Develop mechanisms for recognizing sentiment attributes using domain-specific knowledge.
  • The authors also implemented a prototype pros/cons identification system as a search engine-style interface and an "add-on" for PDF viewers, suggesting practical applications for their work.

Summary updated

Abstract

This paper explores a task for extracting a technological expression and its pros/cons from computer science papers. We report ongoing efforts on an annotated corpus of pros/cons and an analysis of the nature of the automatic extraction task. Specifically, we show how to adapt the targeted sentiment analysis task for pros/cons extraction in computer science papers and conduct an annotation study. In order to identify the challenges of the automatic extraction task, we construct a strong baseline model and conduct an error analysis. The experiments show that pros/cons can be consistently annotated by several annotators, and that the task is challenging due to domain-specific knowledge. The annotated dataset is made publicly available for research purposes.
BibTeX
@misc{shirai2019annotating,
  author = {Hono Shirai and Naoya Inoue and Jun Suzuki and Kentaro Inui},
  title = {{Annotating with Pros and Cons of Technologies in Computer Science Papers}},
  year = {2019},
  doi = {10.18653/v1/w19-2605},
}
Figure