Non-canonical language is not harder to annotate than canonical language

Barbara Plank, Héctor Martínez Alonso, Anders Søgaard

TL;DR

This paper challenges the common assumption that annotating "non-canonical" language (like social media text) is inherently harder than annotating "canonical" language (like newswire). Through a small-scale annotation study, the authors found that annotator agreement was surprisingly similar across domains, suggesting the perceived difficulty might be more about historical resource availability than inherent linguistic complexity.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), there's a prevailing notion that "canonical" language, typically represented by newswire and textbook examples, is easier to process and annotate than "non-canonical" language found in sources like social media. This prejudice often leads to a focus on newswire for building linguistic resources, potentially leaving other, more frequently used forms of language underrepresented. The authors question this assumption, proposing that the difficulty in processing non-canonical language might stem from a lack of resources rather than the language itself being intrinsically more complex to annotate. They aim to investigate whether annotating a specific linguistic task (identifying main verbs and their arguments) is indeed harder for non-canonical text compared to canonical text.

Methods

The researchers conducted a pilot annotation study involving two expert annotators. They were tasked with identifying main verbs (MV) and their arguments (A0, A1) in sentences from five different domains. Crucially, the annotators were given minimal guidelines, aiming to isolate the inherent difficulty of the task itself rather than the annotators' interpretation of complex instructions. The annotation task involved identifying:

  • MAINVERB (MV): The main lexical verb of the predicate.
  • A0: The subject of the verb.
  • A1: The direct object or attribute.

The study focused on comparing agreement rates between the two annotators across these domains.

Datasets

The study used sentences from five corpora, chosen to represent varying degrees of perceived canonicity:

  • Wall Street Journal (WSJ): Considered the most canonical.
  • Answers: Yahoo! Answers (social media).
  • Spoken: Switchboard corpus (spoken language).
  • Fiction: Brown corpus (edited literary text).
  • Twitter: Tweebank (social media).

For each domain, 50 sentences were randomly selected and doubly annotated, resulting in a total of 250 annotated sentences.

Key Insights and Ideas

  • The "Canonical" Bias is Problematic: The authors argue that defining "canonical" language based on newswire is a biased sample of actual language use. Most of the language we produce and consume daily is far more diverse.
  • Annotation Difficulty is Not Domain-Specific (in this study): The pilot study's results suggest that the difficulty of annotating main verbs and arguments is surprisingly similar across canonical (Newswire) and non-canonical (Twitter, Spoken, Fiction, Answers) domains.
  • Resource Skewness is a Major Factor: The historical focus on newswire for NLP resources is likely a significant reason why tools perform poorly on social media, not necessarily because social media language is inherently harder to annotate.
  • Disagreement Reasons May Differ: While overall agreement might be similar, the reasons for disagreement between annotators on social media might differ from those on newswire, potentially involving more ad hoc conventions or different linguistic phenomena.

Key Results

  • Similar Agreement Scores: Label-wise and micro-averaged F1 scores between annotators were comparable across all five domains. Newswire showed surprisingly low agreement, while Answers (social media) was found to be easier.
  • Sentence Length Correlation is Low: The correlation between sentence length and sentence-wise agreement was found to be low (0.1364), suggesting sentence length isn't the primary driver of annotation difficulty.
  • Parser Confidence vs. Agreement: The correlation between parser confidence and annotator agreement varied. Newswire had high parser confidence but moderate agreement, while Answers had high agreement but lower parser confidence, hinting at different underlying challenges.
  • Disagreement Sources: Common sources of disagreement included identifying the main predicate, handling complex sentence structures (like multiple clauses in tweets), and dealing with tokenization issues (e.g., hyphenated words).

Limitations and Future Work

  • Small-Scale Study: The authors explicitly state this is a "severely underpowered pilot annotation study" with only two annotators and 50 sentences per domain. The results should be taken with a grain of salt.
  • Minimal Guidelines: While this was intentional to isolate task difficulty, it might not reflect real-world annotation scenarios where more detailed guidelines are provided.
  • Future Work: The authors emphasize the need to validate these findings on a larger scale with more annotators and a wider range of tasks. They also suggest further research into why social media annotation might be difficult (even if not inherently harder than newswire) and what specific linguistic phenomena contribute to disagreements.

Summary updated

BibTeX
@inproceedings{plank2015non,
  author = {Barbara Plank and Héctor Martínez Alonso and Anders Søgaard},
  title = {{Non-canonical language is not harder to annotate than canonical language}},
  booktitle = {LAW @ NAACL},
  year = {2015},
  url = {https://openreview.net/forum?id=Og9cVVblRSA},
}

Loading related content...

Figure