Annotating Claims in the Vaccination Debate

Benedetta Torsi, Roser MoranteEMNLP 2018

TL;DR

This paper tackles the challenge of automatically identifying argumentative claims within the complex and diverse landscape of online texts, specifically focusing on the vaccination debate. By iteratively simplifying annotation schemes and conducting rigorous experiments, the authors developed a more feasible approach to claim detection, achieving a satisfactory inter-annotator agreement of 57.3% on a newly annotated corpus.

What Problem Does This Paper Address?

Understanding public opinion and identifying misinformation are crucial, especially on topics with significant societal impact like vaccination. Online platforms (social media, forums, blogs) are rich sources of this information, but they present a significant challenge for traditional argumentation mining techniques. Unlike structured texts like essays or editorials, web data is heterogeneous in genre, style, and argumentation structure, making existing annotation schemes ill-suited. The authors aim to adapt and simplify argumentation annotation schemes to effectively capture claims within this noisy web data, with the ultimate goal of analyzing public discourse and potentially detecting misinformation.

Methods

The paper describes an iterative process of developing and refining an annotation scheme for argument components.

  • Initial Complex Schemes: The researchers began by testing two established argumentation schemes:
  • One based on Stab and Gurevych (2017), which identifies major claims, claims, and premises, suitable for argumentative essays.
  • Another based on the Toulmin model (Habernal and Gurevych, 2017), which includes components like claim, grounds, backing, qualifiers, rebuttal, and refutation, designed for web data.
  • Pilot Studies & Simplification: Two pilot studies were conducted using these schemes. The results showed low inter-annotator agreement (IAA), indicating that the complexity and vagueness of the schemes were not suitable for the heterogeneous web corpus. This led to a significant simplification.
  • Final Simplified Scheme: The final experiment focused solely on annotating "claims." The definition of a claim was broadened to include "claim-like statements" that express the author's stance, are refutable, and can be implicitly or explicitly attributed to a source. This simplified approach aimed to make the task more feasible and less time-consuming.

Datasets

  • Vax Corpus: A newly created corpus of 294 documents (approximately 445,574 words) automatically downloaded from the internet, specifically related to the vaccination debate. This corpus is characterized by its heterogeneity in length, genre, and style, including blog posts, editorials, news articles, and science articles.
  • VaxClaim Corpus: The Vax Corpus, annotated using the simplified claim-focused scheme, which will be made publicly available.

Key Insights and Ideas

  • Web Data Demands Simplicity: The core insight is that complex, fine-grained argumentation annotation schemes, while effective for structured texts, are not directly applicable to the messy, heterogeneous nature of web data. Simplification is key to achieving reliable annotations.
  • Claims as the Core: Focusing on "claims" as the central component of an argument proved to be a more manageable and fruitful approach for web data. This allows for capturing the essence of people's stances and opinions.
  • Challenges in Defining "Claim": Even with a simplified definition, annotating claims remains challenging. Issues like "debatability" (how controversial a statement needs to be), "attributability" (who the claim is from), "relatedness to topic," and "granularity" (how to segment claims and their sources) consistently led to disagreements.
  • The "Why" Matters: The paper highlights that understanding why people hold certain stances (their beliefs and attitudes) is crucial, and even statements indirectly related to the topic can be valuable for this analysis.

Key Results

  • Pilot Study 1 (Stab & Gurevych based): Low IAA, particularly for "premise" (0.0%). Major claims had higher agreement (66.7%), but overall, the scheme was not suitable.
  • Pilot Study 2 (Habernal & Gurevych based): Overall IAA of 19.9%. Refutation had the lowest agreement (10.0%), and backing the highest (25.8%). The task was also found to be time-consuming.
  • Final Annotation Task (Simplified Claim Focus): Achieved an overall IAA of 54.4%, with "Claim" component IAA at 57.3%. This is a significant improvement over the pilot studies and is considered satisfactory. The task was completed in 33.5 hours for 100 files.

Limitations and Future Work

  • Limitations:
    • The simplified scheme still faced challenges with annotator disagreement, particularly concerning debatability, attributability, relatedness to the topic, and granularity of sources.
  • The definition of "claim" still requires refinement, especially regarding statements that are not directly related to the topic but reveal user beliefs.
  • The current guidelines lack explicit instructions for handling polarizing claims and speculative statements.
  • Future Work:
    • Develop a more refined definition of "claim" based on the identified error sources.
  • Create more annotated examples and more restrictive guidelines to address specific disagreement patterns.
  • Test the applicability of the simplified scheme to other online debate domains.
  • Develop an automated claim detection system to further measure the task's difficulty.

Summary updated

BibTeX
@inproceedings{torsi2018annotating,
  author = {Benedetta Torsi and Roser Morante},
  title = {{Annotating Claims in the Vaccination Debate}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure