Grammar-based Data Augmentation for Low-Resource Languages: The Case of Guarani-Spanish Neural Machine Translation

Agustín Lucas, Alexis Baladón, Victoria Pardiñas, Marvin M. Agüero-Torales, Santiago Góngora, Luis ChiruzzoNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of low-resource language machine translation by proposing a novel data augmentation technique. They leverage a grammar-based system to generate synthetic parallel data for Guarani-Spanish, demonstrating that pre-training MT models with this synthetic data significantly improves translation performance, even outperforming existing baselines and Google Translate in some directions.

What Problem Does This Paper Address?

Many of the world's languages, despite having millions of speakers, are considered "low-resource" in the field of Natural Language Processing (NLP). This means there's a critical lack of data to train powerful machine translation (MT) systems, creating a vicious cycle: you need tools to collect data, but you need data to build tools. Guarani, an indigenous language spoken by nearly 10 million people, exemplifies this problem. Existing MT systems for Guarani-Spanish struggle due to limited parallel corpora, which are essential for training modern neural MT models. This paper aims to break this cycle by creating a new source of training data through a grammar-based approach.

Methods

The core of this work lies in a novel data augmentation strategy. The authors developed two simple grammars, one for Spanish and one for Guarani, inspired by HPSG formalisms. These grammars are used to generate short, syntactically correct Spanish sentences. A set of "transfer rules" then converts these Spanish sentence structures and their associated parse trees into Guarani. This process involves:

  1. Lexicon Building: Creating a bilingual Guarani-Spanish lexicon by combining existing dictionaries and linguistic resources, annotating words with morphological information.
  2. Synthetic Text Generation: Using the grammars to generate Spanish sentences and their corresponding parse trees. These are then transformed into Guarani sentences using the transfer rules. This method can handle linguistic nuances like Guarani's nasal/oral agreement and Spanish's prepositional phrases.
  3. Synthetic Corpus Creation: The grammar-based generation process yields a "Synthetic Grammar corpus." Additionally, they created a "Synthetic AnCora corpus" by applying the same grammar-based transfer to the Spanish AnCora corpus, aiming for more grounded, real-world text.
  4. Pretraining and Fine-tuning: The generated synthetic data is used to pretrain neural MT models (both seq2seq and transformer architectures) before fine-tuning them on the existing Jojajovai dataset. This pretraining strategy is explored with different combinations of synthetic data and varying numbers of pretraining epochs.

Datasets

  • Jojajovai Corpus: The primary benchmark dataset for Guarani-Spanish MT, used for fine-tuning and evaluation.
  • Synthetic Grammar Corpus: A newly generated corpus of 277,842 sentence pairs created using the authors' grammar-based system.
  • Synthetic AnCora Corpus: Another newly generated corpus of 14,120 sentence pairs, created by applying the grammar-based transfer to the Spanish AnCora corpus.
  • Bible Dataset: A widely used dataset in low-resource MT, also included in experiments.

Key Insights and Ideas

  • Grammar as a Data Generator: The paper demonstrates that a well-defined grammar, even a simplified one, can be a powerful tool for generating synthetic parallel data for low-resource languages, bypassing the need for large existing corpora or pre-trained models for the initial data generation.
  • Pretraining with Synthetic Data Boosts Performance: Pretraining MT models with the synthetically generated data significantly improves translation quality, even outperforming models trained solely on existing data. This suggests that synthetic data can effectively "teach" the model about the language's structure and vocabulary.
  • Synthetic AnCora's Effectiveness: The "Synthetic AnCora" corpus, derived from real-world Spanish text, proved particularly effective, achieving strong results with a relatively small size. This indicates that grounding synthetic data in existing linguistic resources can be highly beneficial.
  • Seq2Seq Models Still Competitive: While transformers are the current state-of-the-art, seq2seq models, especially when tuned and pre-trained with synthetic data, showed competitive performance, sometimes even surpassing transformers in this low-resource setting.

Key Results

  • The best performing models (seq2seq architecture, pre-trained on a combination of synthetic data and the Bible, then fine-tuned on Jojajovai) outperformed previous Jojajovai benchmark baselines.
  • These models also surpassed Google Translate for the Spanish-to-Guarani (es→gn) direction and showed competitive performance for the Guarani-to-Spanish (gn→es) direction.
  • Models pre-trained solely on the Synthetic AnCora corpus achieved near-best results, highlighting its efficiency.
  • The synthetic text generation achieved a structural correctness score of 87.1% from a native speaker's evaluation, though word choice and fluency were lower, which is acceptable for pretraining purposes.

Limitations and Future Work

  • Linguistic Quality of Synthetic Text: The authors acknowledge that the synthetic grammar's linguistic quality is not perfect, with some words not recognized by native speakers and occasional unnatural phrasing. They suggest that improving the grammar could further enhance performance.
  • Decision to Keep Spanish Words: The strategy of retaining Spanish words when no Guarani translation was found in the AnCora corpus was not extensively tested against alternatives.
  • Grammar Complexity vs. Effort: Further research is needed to determine if investing more effort into building a more complex grammar would yield proportionally better pretraining results.
  • Framework Comparison: All experiments were conducted using MarianNMT. Comparing results with other frameworks like OpenNMT could provide a more comprehensive understanding of the impact of the synthetic data.
  • Alternative Data Augmentation: The paper did not explore combining their grammar-based approach with other data augmentation techniques like back-translation, which could be a promising avenue.

Summary updated

BibTeX
@inproceedings{lucas2024grammar,
  author = {Agustín Lucas and Alexis Baladón and Victoria Pardiñas and Marvin M. Agüero-Torales and Santiago Góngora and Luis Chiruzzo},
  title = {{Grammar-based Data Augmentation for Low-Resource Languages: The Case of Guarani-Spanish Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.354},
}
Figure