Multilingual Grammatical Error Annotation: Combining Language-Agnostic Framework with Language-Specific Flexibility

Mengyang Qiu, Tran Minh Nguyen, Zihao Huang, Zelong Li, Yang Gu, Qingyu Gao, Siliang Liu, Jungyeul Park

TL;DR

This paper introduces a new, flexible framework for annotating grammatical errors across many languages, addressing the limitations of existing tools like errant when dealing with linguistic diversity. By combining a core language-agnostic approach with adaptable language-specific extensions and leveraging the stanza NLP library, they enable more consistent and interpretable GEC evaluation for a wider range of languages.

What Problem Does This Paper Address?

Grammatical Error Correction (GEC) is crucial for improving writing, especially for language learners. A key bottleneck in GEC research is the accurate and consistent annotation of errors, which then informs system development and evaluation. While the errant toolkit has been a de facto standard for English GEC, its extension to typologically diverse languages has been challenging. Existing multilingual adaptations often suffer from inconsistent annotation labels, varying preprocessing tools, and a lack of fine-grained error granularity, making cross-lingual comparisons difficult and hindering the development of robust multilingual GEC systems.

Methods

The authors propose a two-tiered approach to multilingual GEC annotation:

  1. Language-Agnostic Foundation: This core layer utilizes the widely adopted MRU (Missing, Replacement, Unnecessary) framework for error classification. This ensures consistency in the fundamental error operations across all languages.
  2. Language-Specific Extensions: This layer allows for tailored annotations to capture unique linguistic features. They introduce structured templates for common error types (e.g., spelling, word order, word boundary) that can be reused or adapted.

A key technical contribution is the reimplementation of errant using the stanza NLP library. stanza provides a consistent, UD-based preprocessing pipeline (tokenization, POS tagging, dependency parsing) for over 70 languages, offering a more stable foundation for multilingual annotation compared to spaCy, which errant originally used. The framework is designed to be modular, allowing researchers to swap out preprocessing tools or add custom error categories as needed.

Datasets

The paper demonstrates the framework's applicability on several languages, including:

  • English: Used for reimplementing and validating the core framework against existing errant results.
  • German, Czech: Evaluated using existing datasets and annotation schemes from prior work (Boyd, 2018; Náplava et al., 2022).
  • Korean: Refinements were made using data from the National Institute of Korean Language (NIKL) corpus.
  • Chinese: Utilized the Chinese GSD treebank, with a focus on adapting segmentation strategies.

No new benchmark datasets were introduced, but the work aims to standardize annotation practices for existing and future multilingual GEC datasets.

Key Insights and Ideas

  • The Power of a Unified Preprocessing Pipeline: Using stanza across languages provides a consistent linguistic analysis foundation, reducing variability that arises from using different tokenizers and POS taggers for each language. This is crucial for reliable cross-lingual evaluation.
  • Modularity is Key for Multilingualism: The two-tiered approach (agnostic core + flexible extensions) is essential. It allows for broad applicability while accommodating the nuances of diverse languages without requiring a complete overhaul for each.
  • Beyond MRU: Fine-Grained Error Types Matter: While MRU provides a good starting point, the paper shows the value of adding language-specific error types (e.g., distinguishing phonetic vs. shape-based spelling errors, or specific word order issues) for more interpretable analysis.
  • Segmentation is a Major Hurdle for some Languages: For languages like Chinese, where word boundaries are not explicit, the choice of segmentation strategy (e.g., LTP vs. GSD) significantly impacts error annotation. The framework's ability to integrate different segmentation schemes is a critical feature.

Key Results

  • The reimplemented errant using stanza for English achieved comparable results to the original errant on the T5 GEC system outputs, demonstrating the validity of the core reimplementation (Table 1 shows F0.5 scores of 0.5501 for original errant and 0.5503 for their implementation).
  • For European languages (German, Czech), the framework successfully generated generic error annotations using the MRU scheme and stanza's universal POS tags, showing consistency with prior work's operation counts (Table 2).
  • For Korean, minor refinements to word boundary and spelling error categories were implemented, demonstrating targeted customization.
  • For Chinese, the framework showed improved annotation consistency by integrating an LTP-style segmentation approach, leading to more accurate edit spans and error categories, especially for compound expressions (Figure 6 illustrates this).

Limitations and Future Work

  • Annotation Improvements Analysis: The paper acknowledges that a detailed analysis of how their framework improves annotation quality (beyond consistency) and its downstream effects on GEC system performance is left for future work.
  • NLP Tool Limitations: While stanza is powerful, it's not explicitly optimized for noisy or learner-generated text, which could introduce variability in edge cases, especially for languages with complex morphosyntax or ambiguous segmentation.
  • Mapping Representations: Future work could explore methods to map between different annotation representations to improve compatibility and facilitate comparative evaluations.
  • Expanding Language Coverage: The framework's design, leveraging stanza, makes it readily applicable to many more languages, which is a clear future direction.

Summary updated

Abstract

Grammatical Error Correction (GEC) relies on accurate error annotation and evaluation, yet existing frameworks, such as $\texttt{errant}$, face limitations when extended to typologically diverse languages. In this paper, we introduce a standardized, modular framework for multilingual grammatical error annotation. Our approach combines a language-agnostic foundation with structured language-specific extensions, enabling both consistency and flexibility across languages. We reimplement $\texttt{errant}$ using $\texttt{stanza}$ to support broader multilingual coverage, and demonstrate the framework's adaptability through applications to English, German, Czech, Korean, and Chinese, ranging from general-purpose annotation to more customized linguistic refinements. This work supports scalable and interpretable GEC annotation across languages and promotes more consistent evaluation in multilingual settings. The complete codebase and annotation tools can be accessed at https://github.com/open-writing-evaluation/jp_errant_bea.
BibTeX
@misc{qiu2025multilingual,
  author = {Mengyang Qiu and Tran Minh Nguyen and Zihao Huang and Zelong Li and Yang Gu and Qingyu Gao and Siliang Liu and Jungyeul Park},
  title = {{Multilingual Grammatical Error Annotation: Combining Language-Agnostic Framework with Language-Specific Flexibility}},
  year = {2025},
  eprint = {2506.07719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.07719},
}
Figure