Stating with Certainty or Stating with Doubt: Intercoder Reliability Results for Manual Annotation of Epistemically Modalized Statements

Victoria L. RubinNAACL 2007

TL;DR

This paper investigates how reliably humans can identify and categorize different levels of certainty expressed in written news. The findings reveal that distinguishing subtle shades of certainty is highly subjective, posing a significant challenge for automated systems aiming to extract opinion and sentiment from text.

What Problem Does This Paper Address?

In natural language processing, understanding the nuances of how a writer expresses their confidence in a statement is crucial for tasks like opinion extraction and sentiment analysis. While we intuitively understand that some statements are presented as absolute facts and others as mere possibilities, precisely quantifying these "shades of certainty" in text is difficult. This paper tackles the challenge of manually annotating these epistemically modalized statements, specifically focusing on the intercoder reliability – how consistently different human annotators agree on the certainty level of a given statement. The authors highlight the lack of standardized methods and discrete categories for this task in computational linguistics, which hinders the development of robust automated systems.

Methods

The study employs a manual annotation approach based on a conceptual categorization model developed by Rubin (2006). This model defines five levels of certainty: ABSOLUTE, HIGH, MODERATE, LOW CERTAINTY, and UNCERTAINTY. These levels are further contextualized by three dimensions: PERSPECTIVE (who is expressing the certainty – the writer, a participant, or an expert), FOCUS (whether it's about facts or opinions/judgments), and TIME (past, present, or future).

Two experiments were conducted:

  • Experiment 1: Three coders were trained for one hour and used a general codebook.
  • Experiment 2: A fourth annotator underwent a more intensive five-hour training with a revised, more rigidly specified codebook and a keyword index.

Intercoder agreement was measured using percent agreement and Cohen's kappa statistic for:

  1. Identifying sentences with explicit certainty markers.
  2. Identifying the specific certainty markers within those sentences.
  3. Classifying the markers into the defined certainty levels and contextual dimensions.

Datasets

The dataset consisted of 80 randomly selected articles from the AQUAINT Corpus of English Texts, distributed by The New York Times Services in 2000. This corpus comprised 2,243 sentences, with 866 from editorials and 1,377 from news reports. A subset of 10 articles (272 sentences, approximately 12% of the full dataset) was used for the annotation experiments.

Key Insights and Ideas

  • Subjectivity of Certainty: The most striking insight is that even with detailed instructions, human annotators struggle to consistently agree on the precise level of certainty expressed in a statement. This suggests that the boundaries between different shades of certainty are inherently subjective.
  • Complexity of English: English, despite its rich inventory of modal expressions, may not be precise enough to reliably distinguish multiple subtle epistemic shades. This makes it difficult for both humans and machines to categorize them consistently.
  • Context Matters: The pragmatic context (perspective, focus, and time) significantly influences how certainty is perceived and interpreted, adding another layer of complexity to annotation.
  • Potential for Simplification: The findings raise questions about the utility of a five-level certainty scale, suggesting that a simpler binary distinction between "certainty" and "doubt" might be more achievable and practical for automated systems.

Key Results

  • Experiment 1:
    • Coders agreed on whether a sentence contained an explicit certainty marker 71% of the time (Cohen's kappa = 0.33).
  • Agreement on the specific certainty markers within identified sentences was 54%.
  • Agreement on the categorization of markers into levels and dimensions was generally low, with slight agreement (kappa 0.13-0.15) for level and focus, and fair agreement (kappa 0.41-0.44) for perspective and time.
  • Experiment 2:
    • Agreement on explicit certainty markers improved to 67%.
  • Overall ability to distinguish certainty-qualified from unmarked statements showed better agreement (kappa 0.51).
  • The perspective dimension showed a relatively intuitive categorization (kappa 0.65).
  • However, agreement on the certainty level itself remained problematic (kappa 0.41), and time categorization continued to be a challenge (kappa 0.31). Distinguishing facts from opinions in conjunction with certainty was also difficult (kappa 0.16).

Limitations and Future Work

The authors acknowledge that the intercoder reliability results, while improved in Experiment 2, are still modest, attesting to the inherent complexity of epistemically modalized statements. They suggest that while stricter annotation instructions and longer training can enhance agreement, it's unclear if a five-level distinction is superior to a simpler binary one.

Future work proposed includes:

  • Revising the number of discrete categories on the epistemic continuum.
  • Further refining the conceptual definition of certainty levels.
  • Validating the collection of agreed-upon certainty markers on a much larger dataset.
  • Using the developed typology as input for machine learning algorithms to automate certainty identification and extraction.

Summary updated

BibTeX
@inproceedings{rubin2007stating,
  author = {Victoria L. Rubin},
  title = {{Stating with Certainty or Stating with Doubt: Intercoder Reliability Results for Manual Annotation of Epistemically Modalized Statements}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure