Annotating Decomposition in Time: Three Approaches for Again

Martin Kopf, Remus GergelACL 2023

TL;DR

This paper tackles the challenge of semantically annotating the ambiguous adverb "again" across historical English texts. They introduce three novel methods – expert annotation, machine learning prediction, and informed crowdsourcing – to create a robust dataset and explore efficient annotation strategies for diachronic linguistics.

What Problem Does This Paper Address?

Understanding how language meaning evolves over time (diachronic linguistics) is crucial for linguistic theory. However, creating high-quality annotated datasets for historical texts is a significant bottleneck. This paper focuses on the adverb "again," which has multiple meanings (e.g., repetition vs. restoration of a previous state) and whose usage has shifted historically. The authors aim to address the practical challenges of:

  • Costly extraction: Obtaining and annotating historical data is time-consuming and expensive.
  • Annotation quality and validation: Ensuring consistency and accuracy across annotators is difficult, especially with subjective linguistic phenomena.
  • Scalability: Manually annotating large historical corpora is often infeasible.

They propose methods to overcome these hurdles, making diachronic semantic analysis more accessible and scalable.

Methods

The paper presents three distinct approaches to annotating "again":

  1. Exhaustive Expert Annotation: This is the "gold standard" method. A team of expert annotators meticulously analyzed uses of "again" in syntactically parsed historical English corpora, classifying its readings and marking relevant linguistic elements. This process involved developing detailed guidelines and iterative refinement.
  2. Machine Learning Prediction (Multinomial Naïve Bayes): To explore automation, the authors trained a Naïve Bayes classifier to predict the readings of "again." They conducted a meta-analysis to identify the most effective features for this task, drawing from semantic annotations, syntactic parsing, and surface text features.
  3. Informed Crowdsourcing: Recognizing the limitations of expert annotation for large-scale projects, they designed an "informed crowdsourcing" experiment. This involved recruiting university students (English majors) who received condensed guidelines and training. This approach aims to leverage a larger pool of annotators while mitigating the lack of true native speaker intuition for historical language.

Datasets

The primary dataset used is the Penn Parsed Corpus of Modern British English (PPCMBE2), specifically focusing on uses of "again" from the last two to four centuries. The authors also mention using data from the Penn-Helsinki Parsed Corpus of Early Modern English (PPCEME).

A key contribution is the creation of a newly annotated dataset of 1,901 uses of "again" from PPCMBE2, which they make available. This dataset serves as the "gold standard" against which their other methods are evaluated.

Key Insights and Ideas

  • "Again" has evolved: The frequency of different readings of "again" has shifted significantly over time, with a notable decrease in "restitutive/counterdirectional" uses in later periods.
  • Feature engineering is key for ML: For machine learning approaches, a combination of semantic, syntactic, and surface-level features is crucial for accurate prediction of "again" readings. Surprisingly, a smaller set of core features can achieve high accuracy, and adding too many features can sometimes hinder performance.
  • Crowdsourcing can be effective with guidance: "Informed crowdsourcing" can yield valuable annotations, especially when participants are given specific training and clear guidelines, even if they aren't seasoned experts.
  • Focusing on difficult cases: The "restitutive/counterdirectional" (res/ct) readings of "again" are more prone to annotation errors and confusion, suggesting that expert review should prioritize these cases to improve overall annotation quality efficiently.
  • Crowd confidence matters: Unit Quality Scores (UQS) indicate that crowd annotators are more confident in their classifications of "repetitive" readings than "restitutive" ones, aligning with the confusion matrix results.

Key Results

  • Expert Annotation: The paper details the process of creating a gold standard dataset of 1,901 "again" instances.
  • Machine Learning: The Multinomial Naïve Bayes classifier achieved an average accuracy of up to 81.46% in classifying "again" as either "repetitive" (rep) or "restitutive/counterdirectional" (res/ct). A core set of 5 features yielded an accuracy of 80.67%.
  • Crowdsourcing:
    • KMeans clustering of crowd annotations achieved the highest accuracy rates among the crowdsourcing approaches.
  • Overall accuracy for KMeans clustering was 84.1% (based on the gold standard).
  • Accuracies for "repetitive" readings were consistently high (around 89.8%), while "restitutive/counterdirectional" readings showed lower accuracy, particularly in older data (75.0%).
  • A confusion matrix revealed that 19.8% of crowd annotations were inaccurate, with most errors occurring in distinguishing "repetitive" and "restitutive" readings.

Limitations and Future Work

  • Gold Standard Imperfection: The authors acknowledge that even their expert annotation process is subject to ongoing improvement and may not be perfectly exhaustive.
  • Machine Learning Limitations: The Naïve Bayes classifier is sensitive to feature selection, and the current model does not account for word order or feature weights, suggesting potential for improvement by incorporating more sophisticated models or feature types (e.g., bigrams).
  • Crowdsourcing Nuance: While "informed crowdsourcing" shows promise, the lack of true native intuition for historical language remains a fundamental difference from expert annotation. The authors suggest that focusing expert review on areas of crowd disagreement (particularly for res/ct readings) can enhance robustness.
  • Future Work: The authors suggest expanding the feature set for machine learning, exploring alternative crowdsourcing strategies, and applying these methods to other decompositional markers or low-frequency phenomena in diachronic linguistics. They also highlight the potential for these methods in scenarios where syntactic parsing is unavailable.

Summary updated

BibTeX
@inproceedings{kopf2023annotating,
  author = {Martin Kopf and Remus Gergel},
  title = {{Annotating Decomposition in Time: Three Approaches for Again}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure