A Word-and-Paradigm Workflow for Fieldwork Annotation

Maria Copot, Sara Court, Noah Diewald, Stephanie Antetomaso, Micha ElsnerACL 2022

TL;DR

This paper introduces a novel Word-and-Paradigm (WP) based workflow for morphological annotation in linguistic fieldwork, aiming to accelerate the process and increase community involvement. By leveraging unsupervised learning and a relational approach, it reduces the need for early, potentially incorrect, segmentation and feature labeling, allowing non-linguist native speakers to contribute effectively.

What Problem Does This Paper Address?

Linguistic fieldwork, especially for endangered languages, faces significant challenges in morphological annotation. Traditional methods often rely on segmenting words into morphemes and assigning labels, which is time-consuming, requires extensive linguistic training, and can lead to premature, potentially incorrect, assumptions about a language's structure. This is particularly problematic in low-resource settings where data is scarce and understanding is nascent. The paper addresses the need for a more efficient, flexible, and accessible annotation process that can be readily adopted by both linguists and native speakers, even those without formal linguistic training.

Methods

The proposed workflow is grounded in Word-and-Paradigm (WP) morphology, which treats words as the primary unit of analysis and focuses on the relationships between them. The core of their approach involves:

  • Unsupervised Paradigm Discovery: They utilize a system (based on SIGMORPHON 2020 Task 2) that takes raw text and a list of target lemmas to identify potential inflected forms and cluster them into paradigm cells without prior segmentation. This initial step provides a starting point for annotation.
  • Concordance-Style Datasets: The output of the unsupervised model is used to generate datasets where potential forms are presented with their surrounding text context. This allows annotators to assess the correctness of the proposed groupings based on usage.
  • Iterative Refinement: Annotators correct errors in the model's output, identifying incorrect forms within proposed paradigms and suggesting missing forms. This feedback loop is intended to bootstrap more accurate analyses.
  • Active Learning (Proposed): While not fully implemented in this proof-of-concept, the workflow is designed to integrate active learning, where the model would query annotators on the most uncertain distinctions to further improve efficiency and accuracy.

Datasets

The paper uses existing datasets for evaluation:

  • English: The GUM treebank (Zeldes, 2017)
  • Croatian: The SETimes treebank (Agić and Ljubešić, 2015)

For the Wao Terero experiment, they used the New Testament as the raw text corpus due to the lack of readily available, open-access, and consistently orthographed corpora. They acknowledge that the New Testament can be atypical and stylized.

Key Insights and Ideas

  • WP as a Bridge: The Word-and-Paradigm approach naturally aligns with computational methods and offers a more flexible starting point for annotation than traditional Item-and-Arrangement methods, deferring difficult segmentation and labeling decisions.
  • Context is King: Presenting potential forms within their original sentence context is crucial for annotators, especially non-linguists, to make informed decisions about correctness and belonging.
  • Community Empowerment: The workflow's design, particularly its reliance on relational patterns rather than technical jargon, significantly lowers the barrier to entry for native speakers, fostering greater community engagement in language documentation.
  • Speed vs. Accuracy Trade-off: The experiments highlight that even with limited time and training, annotators can significantly improve upon unsupervised model outputs, demonstrating the value of human-in-the-loop annotation.
  • Modular Design: The workflow's modularity allows for easy integration of future advancements in computational linguistics, making it adaptable and future-proof.

Key Results

  • Annotation Speed: English annotators processed an average of 444 lexeme tokens (including regex searches) in 30 minutes, while Croatian annotators processed 306. Cell annotation in English was rapid and highly accurate.
  • Accuracy Improvement: Annotators were capable of reliably rejecting incorrect forms proposed by the model. For instance, in English lexeme evaluation, annotators achieved accuracies of 84% and 83% compared to the baseline of 81%. Similar improvements were seen in cell evaluations.
  • Wao Terero Engagement: Wao Terero speakers, with only 10 minutes of training, were able to participate in the annotation task, finding it valuable and not dull, despite some challenges with the data. This demonstrates the potential for non-linguist participation.
  • Regex Limitations: Regular expression searches, while used, did not significantly contribute to finding new forms, suggesting a need for more interactive search tools.

Limitations and Future Work

  • Data Availability: The reliance on existing corpora for English and Croatian, and the use of a potentially atypical corpus for Wao Terero, highlights the ongoing challenge of data availability for low-resource languages.
  • Ambiguity and Agreement: The Wao Terero experiment revealed instances where the linguist's annotations were the complement of a speaker's, indicating that ambiguity in the data can affect annotator agreement. Future work could explore heuristics or methods to address such ambiguities.
  • Interactive Environment: The authors emphasize that this is a proof-of-concept for the initial stage. The full proposed workflow envisions an interactive environment that would allow annotators to view proposed paradigm tables alongside text, search more effectively, and potentially improve recall errors.
  • Active Learning Integration: The full integration of active learning is a key next step to further optimize the annotation process by focusing annotator effort on the most uncertain distinctions.

Summary updated

BibTeX
@inproceedings{copot2022word,
  author = {Maria Copot and Sara Court and Noah Diewald and Stephanie Antetomaso and Micha Elsner},
  title = {{A Word-and-Paradigm Workflow for Fieldwork Annotation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure