System Description of the NordicsAlps Submission to the AmericasNLP 2024 Machine Translation Shared Task

Joseph Attieh, Zachary William Hopton, Yves Scherrer, Tanja SamardzicNAACL 2024

TL;DR

This paper explores the impact of tokenization strategies on machine translation (MT) for low-resource indigenous languages. The NordicsAlps team found that a novel "redundancy-driven" subword tokenization method significantly outperformed standard approaches, achieving top rankings in the AmericasNLP 2024 shared task. This highlights the critical role of careful tokenization in maximizing data efficiency for resource-constrained MT.

What Problem Does This Paper Address?

Machine translation (MT) systems for indigenous languages of the Americas face significant challenges due to their low-resource nature. These challenges include a scarcity of parallel data, non-standardized orthographies, and a general lack of language technology. While cross-lingual transfer and data-efficient approaches are crucial, the optimal configuration of MT systems, particularly concerning tokenization, remains an open question. Tokenization, the process of breaking down text into smaller units, directly impacts vocabulary size and, consequently, model efficiency and performance. Previous work has often relied on arbitrary vocabulary sizes (e.g., 32k tokens) or standard subword tokenization methods like Byte Pair Encoding (BPE). This paper investigates whether a more principled approach to tokenization, specifically one that considers linguistic redundancy, can improve MT performance in these challenging low-resource scenarios.

Methods

The NordicsAlps team explored two main tokenization strategies:

  • Byte-level Tokenization: This approach maps text directly to bytes (using UTF-8 encoding), resulting in a very small, fixed vocabulary of 256 tokens. This eliminates the need for explicit text preprocessing and subword segmentation. They experimented with two byte-level architectures: Byte-SEMD (single encoder, multiple decoders) and Byte-SESD (single encoder, single shared decoder).
  • Redundancy-Driven Subword Tokenization (BPE-MR): This novel approach aims to optimize subword vocabulary size by minimizing "text redundancy." The intuition is that tokenizations that compress text more efficiently (i.e., result in shorter sequences) might lead to better MT performance. This method is inspired by observations that smaller vocabularies (around 2k tokens) and efficient compression correlate with better MT scores. They trained BPE tokenizers to perform a fixed number of merges (300) for each language.

A baseline system (U-SP) using a standard SentencePiece tokenizer with a 32k vocabulary was also trained for comparison. All models were trained using a multilingual one-to-many setup, translating from Spanish to eleven indigenous languages, with English also included as a high-resource target.

Datasets

The paper utilizes data provided by the AmericasNLP 2024 shared task organizers. This includes:

  • Spanish-English Parallel Data: A subset of common sources like Europarl, GlobalVoices, and NewsCommentary, with validation data from the Spanish-English WMT-News corpus.
  • Spanish-Indigenous Language Data: Parallel data for eleven indigenous American languages: Asháninka (cni), Aymara (aym), Bribri (bzd), Chatino (ctp), Guarani (gn), Hñähñu (oto), Nahuatl (nah), Quechua (quy), Raramuri (tar), Shipibo-Konibo (shp), and Wixarika (hch). This also includes extra and synthetic datasets where available.

Notably, the authors did not include Bible data, as previous editions showed it did not improve translation quality, and it was not part of the organizer-provided datasets. Due to time constraints, they also did not apply extensive filtering or cleaning to the parallel data for this year's submission.

Key Insights and Ideas

  • Tokenization is Paramount for Low-Resource MT: The study strongly suggests that the choice of tokenization strategy is not a minor hyperparameter but a critical factor influencing MT performance, especially when data is scarce.
  • Redundancy as a Guiding Principle: The "redundancy-driven" approach (BPE-MR) demonstrated that minimizing text redundancy can lead to more effective subword vocabularies for MT, outperforming standard methods.
  • Byte-Level Models Show Promise but Need More Training: While byte-level tokenization offers a theoretically appealing way to handle arbitrary text without preprocessing, the byte-level models in this study were undertrained, hindering their full potential. This suggests that while promising, these models require more extensive training budgets.
  • Language-Specific Post-Processing Matters: The paper highlights that applying language-specific post-processing rules (e.g., for Guarani and Hñähñu) can significantly boost performance, even after initial MT.

Key Results

The NordicsAlps team submitted three runs: BPE-MR, Byte-SEMD, and Byte-SESD.

  • BPE-MR Dominance: The BPE-MR system achieved the best overall performance, ranking first in average chrF++, chrF, and BLEU scores across all languages. It also secured the top rank for 5 out of 11 languages and was second for 2 others.
  • Competitive Performance: For several languages (Bribri, Asháninka, Hñähñu, Quechua), the competition was extremely close, with top submissions being within one chrF++ point of each other.
  • Baseline Underperformance: The U-SP baseline system with standard SentencePiece tokenization showed much more volatile training curves and occasional language-specific "breakdowns" in performance, leading the authors to not submit it.
  • Byte-Level Undertraining: The byte-level models (Byte-SEMD and Byte-SESD) performed less well, which the authors attribute to undertraining due to their larger batch size requirements and longer training budgets.

Limitations and Future Work

  • Byte-Level Undertraining: The most significant limitation acknowledged is the undertraining of the byte-level models. Future work could involve providing these models with more substantial training resources to assess their true potential.
  • Early Stopping for BPE-MR: For the BPE-MR tokenizers, the early stopping criterion for determining the optimal number of merges was never met within the first 350 merges. Further research is needed to refine this criterion for finding the ideal stopping point.
  • Explaining U-SP Breakdowns: The authors note that they currently lack an explanation for the occasional performance breakdowns observed in the U-SP model for specific languages. Investigating the causes of these fluctuations would be valuable.
  • Further Exploration of Tokenization: The success of BPE-MR suggests that further exploration into redundancy-driven or other linguistically motivated tokenization strategies could yield further improvements for low-resource MT.

Summary updated

BibTeX
@inproceedings{attieh2024system,
  author = {Joseph Attieh and Zachary William Hopton and Yves Scherrer and Tanja Samardzic},
  title = {{System Description of the NordicsAlps Submission to the AmericasNLP 2024 Machine Translation Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure