Reusing a Multi-lingual Setup to Bootstrap a Grammar Checker for a Very Low Resource Language without Data

Inga Lill Sigga Mikkelsen, Linda Wiechetek, Flammie A PirinenACL 2022

TL;DR

This paper presents a novel approach to building grammar checkers for extremely low-resource languages, like Lule Sámi, by leveraging existing tools and resources from a related, better-resourced language (North Sámi). The authors demonstrate a practical workflow that significantly reduces development time and data requirements, enabling the creation of essential language technology for endangered languages.

What Problem Does This Paper Address?

Developing advanced NLP tools, such as grammar checkers, for languages with very few speakers (low-resource languages) is a significant challenge. These languages often lack the large, annotated datasets (gold corpora) that are crucial for training modern machine learning models. This scarcity hinders language maintenance and revitalization efforts, as these tools are vital for ensuring the continued use and accessibility of a language in the digital age. The paper specifically addresses the need for such tools for Lule Sámi, a language with an estimated 800-3,000 speakers, by proposing a method that bypasses the need for extensive data.

Methods

The core of the paper's methodology lies in reusing and adapting existing linguistic resources and infrastructure. The authors utilize a pre-existing multilingual framework called GiellaLT, which provides a foundation for building language technology tools. Their approach involves:

  • Rule-Based System: Instead of machine learning, they employ a rule-based system, combining finite-state automata (FST) for morphological analysis and Constraint Grammar (CG) for syntactic and semantic processing.
  • Leveraging Related Languages: The primary strategy is to reuse components and rules from a closely related language, North Sámi, which already has a well-developed grammar checker. This includes reusing morphological analyzers, tokenizers, and even adapting syntactic rules.
  • Adaptation for Erroneous Input: A key adaptation is modifying existing tools (like the disambiguator) to handle potentially erroneous input, which is common in texts written by learners or those less proficient. This involves relaxing strict rules and paying more attention to misspelled words.
  • Building Regression Tests: Instead of a large gold corpus of errors, the authors focus on creating a smaller set of regression tests. These are test sentences containing specific, frequent error types, which are manually marked and corrected. This allows for iterative development and evaluation.
  • Categorization of Errors: They categorize errors into three main types: phonology/typography-based, morpho-syntactic, and writing convention-based. This systematic approach helps in developing targeted correction rules.

Datasets

  • Lule Sámi Gold Corpus: A corpus of 32,202 words with 3,772 marked writing errors was built in 2013. This corpus was initially intended for spellchecker evaluation but also contains syntactic and morpho-syntactic errors. The paper uses this corpus for testing and improving the grammar checker, acknowledging that it was originally marked primarily for spelling errors.
  • North Sámi Resources: Existing morphological analyzers, tokenizers, and grammar rules from North Sámi were reused and adapted.
  • Regression Tests: A set of manually created test sentences for specific error types was developed as part of the evaluation process.

Key Insights and Ideas

  • Data-Light Development is Possible: The paper strongly demonstrates that high-quality NLP tools can be built for very low-resource languages without relying on massive datasets, by intelligently reusing existing infrastructure and knowledge.
  • Cross-Lingual Transfer is Powerful: Reusing resources from a related language significantly accelerates development and reduces costs. The success with Lule Sámi and North Sámi highlights the potential for this approach in other language families.
  • Focus on Frequent Errors: Prioritizing the development of rules for the most common error types, rather than trying to cover every possible error, is an efficient strategy for bootstrapping a functional grammar checker.
  • Regression Tests as a Development Tool: Creating and using regression tests is an effective way to iteratively develop and refine grammar rules, especially when a large gold corpus is unavailable or not perfectly suited for grammar error analysis.
  • Language Revitalization Impact: By making advanced tools accessible, this approach can directly contribute to the maintenance and revitalization of endangered languages, empowering speakers to use their language more confidently in digital contexts.

Key Results

  • Rapid Development: The Lule Sámi grammar checker was developed in approximately 120 hours of work (40 hours of meetings and 80 hours of individual work), involving two linguists, one of whom is a native speaker.
  • Rule-Based Performance: The system, with 64 rules and 17 rule types, showed promising results in initial evaluations, particularly for relative pronoun and numeral/determiner agreement, and modal verb constructions, achieving precision and recall as high as 98%.
  • Identification of Real Errors: The grammar checker successfully identified real errors in a university-level thesis and a manually proofread Lule Sámi corpus, even finding errors missed by human proofreaders.
  • Iterative Improvement: The process of using the grammar checker on existing corpora and analyzing false positives and negatives proved valuable for refining rules and improving the error mark-up of the gold corpus.

Limitations and Future Work

  • Noise and False Positives: The current version of the grammar checker still produces a significant amount of "noise" with more false positives than true positives, indicating a need to further restrict the rules to improve usability.
  • Adaptation for Learners: While the focus was on high-proficiency writers, the paper acknowledges that texts from language learners often contain different and more complex errors, requiring a distinct grammar checker.
  • Full Evaluation: The paper mentions plans for a full evaluation on the marked-up gold corpus and an independent corpus.
  • Broader Integration: The authors emphasize that the true impact of these tools will be realized when they are integrated into respective text processing programs for the language community.
  • Future Goals: The team aims to release a beta version of the Lule Sámi grammar checker in 2022, with the goal of achieving at least 100 examples per error type for regression tests, with 90% precision and 70% recall, to ensure its usefulness for a wider audience.

Summary updated

BibTeX
@inproceedings{mikkelsen2022reusing,
  author = {Inga Lill Sigga Mikkelsen and Linda Wiechetek and Flammie A Pirinen},
  title = {{Reusing a Multi-lingual Setup to Bootstrap a Grammar Checker for a Very Low Resource Language without Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure