Detecting Annotation Errors in Morphological Data with the Transformer

Ling Liu, Mans HuldenACL 2022 · DOI

Abstract

Morphological datasets are foundational to Natural Language Processing but are prone to human annotation errors. This paper demonstrates that Transformer models can serve as highly effective "spell-checkers" for these complex linguistic tables. By employing a "leave-n-out" training strategy, the authors achieve near-perfect recall in detecting various annotation errors across multiple languages, even when up to 30% of the training data is intentionally corrupted.

Introduction

In the world of Natural Language Processing (NLP), morphological data—the structured rules for word transformations (e.g., walk → walking → walked)—is essential. However, creating these datasets is a manual, expensive process requiring linguistic expertise, making errors like typos or "linguistic confusion" (swapping similar forms) inevitable. While existing error detection often relies on non-neural techniques, this study investigates whether the Transformer architecture can automatically flag suspicious entries, significantly reducing human quality control efforts.

Methodology: The Leave-n-Out Approach

The core of the proposed system is a "leave-n-out" cross-validation strategy. Instead of a single model, the researchers train multiple Transformer models corresponding to the number of slots in a word's paradigm (the table of all its forms).

Visual representation of the leave-n-out cross-validation strategy across different models.

For a paradigm with $m$ slots, $m$ separate models are trained. For example, Model 1 is trained on all slots except the first one across the entire dataset. It is then tasked with predicting the missing first slot. If the model's prediction deviates from the human-provided annotation, that entry is flagged as a potential error. This leverages the Transformer's ability to internalize underlying linguistic patterns and identify anomalies that break those "rules."

Experimental Setup

The study utilized data from the UniMorph project, selecting four languages to represent diverse morphological systems:

  • Finnish: Highly agglutinative with large paradigms.
  • German: Features significant syncretism (identical forms for different slots).
  • Russian: A fusional system using the Cyrillic script.
  • Spanish: Large, generally regular verb paradigms.

To evaluate the system, three types of Artificial Errors were introduced:

  1. Typographic (Type I): Random character-level noise.
  2. Linguistic Confusion (Type II): Swapping forms within the same table.
  3. Self-Adversarial (Type III): Using a Transformer to generate plausible but incorrect forms by picking the second-best beam search result.

Results and Insights

The experimental results highlight the robustness and effectiveness of the Transformer-based approach across various linguistic contexts.

Performance metrics (accuracy, precision, recall, and F1) for Artificial Error III across different languages and noise rates.

Near-Perfect Recall

Across all tested languages and error types, the model achieved a recall near 1.0. This indicates that the system successfully identifies almost every introduced error. In error detection, high recall is critical; it is preferable to review a few false positives than to allow actual errors to remain in the "gold-standard" data.

Robustness to Noise

A key finding was the model's resilience. Accuracy remained high even when the training data was corrupted by up to 30% noise. This suggests that Transformers can effectively learn the dominant linguistic "grammar" even from a flawed "textbook."

Practical Utility

While precision (the ratio of true errors to total flags) was lower when errors were rare (approx. 0.11 at low error rates), it scaled significantly as the error rate increased. Even at a realistic 1% error rate, precision reached approximately 0.43, meaning a human editor would only find about half of the flags to be false alarms—a manageable workload for large-scale data cleaning.

Conclusion

The "predict-and-compare" Transformer approach offers a powerful tool for maintaining the integrity of morphological datasets. While the model struggles with systematic errors (errors repeated consistently across the dataset) and currently requires relatively complete inflection tables, its high recall and robustness to noise make it a valuable asset for linguistic annotation. Future work will explore its applicability to sparse data and other complex linguistic tasks like syntactic parsing.

Summary updated

BibTeX
@inproceedings{liu2022detecting,
  author = {Ling Liu and Mans Hulden},
  title = {{Detecting Annotation Errors in Morphological Data with the Transformer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.19},
}
Figure