Benchmarking Low-Resource Machine Translation Systems

Ana Alexandra Morim da Silva, Nikit Srivastava, Tatiana Moteu Ngoli, Michael Röder, Diego Moussallem, Axel-Cyrille Ngonga NgomoACL 2024

TL;DR

This paper benchmarks four open-source machine translation systems across 26 languages, including low-resource ones, to assess both translation quality and computational efficiency. They found that open-source systems can perform well on low-resource languages, but performance can be inconsistent, and they introduce BENG, a platform for reproducible benchmarking.

What Problem Does This Paper Address?

Machine translation (MT) is crucial for global accessibility, but evaluating new systems, especially for low-resource languages (LRLs), is challenging. Existing studies often compare against a single system or commercial solutions, leaving a unclear picture of the best-performing open-source options. This lack of clarity hinders progress in making MT accessible to the vast majority of the world's languages, which are currently underserved by NLP technologies. The authors aim to provide a comprehensive benchmark of publicly available, open-source MT systems, focusing on both effectiveness and efficiency for LRLs.

Methods

The researchers evaluated four open-source Neural Machine Translation (NMT) systems: LibreTranslate, Opus MT, NLLB, and mBART50. Their evaluation focused on both translation quality (effectiveness) and computational speed (efficiency). They used the NEAMT (Naïve Entity Aware Machine Translation) tool for executing experiments, which allows for modular deployment of MT systems. For evaluation, they employed standard metrics like BLEU, METEOR, chrF++, and TER. The experiments were conducted on four parallel machine-translation benchmark datasets and involved 26 languages, categorized according to their resource availability. The results are made publicly available through BENG, a FAIR (Findable, Accessible, Interoperable, Reusable) benchmarking platform.

Datasets

The study utilized four parallel machine-translation benchmark datasets:

  • OPUS100: A randomly sampled dataset from the OPUS corpus, covering 100 languages, with a focus on English.
  • Europarl: A parallel translation dataset from the European Parliament proceedings, covering 11 languages.
  • IWSLT2017: A parallel dataset based on TED talks, used for the IWSLT 2017 multilingual translation task evaluation, with language pairs from 5 languages.
  • FLORES-200: A manually curated dataset covering 204 languages, designed to be a benchmark for low-resource and multilingual machine translation.

Key Insights and Ideas

  • Open-source systems can perform well on LRLs: Contrary to potential assumptions, the study demonstrates that freely available NMT systems can achieve high BLEU scores even on low-resource languages, highlighting the progress in the NLP community.
  • Performance inconsistency on LRLs: While systems can perform well, their performance on LRLs is often more variable compared to high-resource languages. This suggests that factors beyond just language resource availability influence translation quality.
  • LibreTranslate leads in efficiency: The study found that LibreTranslate exhibited the highest token throughput (efficiency) among the evaluated systems.
  • Dataset-specific performance: Some systems showed proficiency on certain languages within a specific dataset, but this performance did not always translate to other datasets, suggesting potential overfitting or dataset-specific strengths.
  • Language families and performance: Preliminary observations suggest that languages from the same family might exhibit similar performance trends, but this hypothesis requires further investigation with a larger language set.

Key Results

  • NLLB performed best in the FLORES-200 dataset for 20 out of 26 languages with statistically significant differences.
  • Opus MT outperformed other systems in the OPUS100 dataset for 19 out of 26 languages.
  • LibreTranslate showed the best performance in the Europarl dataset.
  • mBART50 achieved the best results in the IWSLT2017 dataset.
  • LibreTranslate had the highest token throughput across most experiments.
  • The study observed increased deviations in BLEU scores for LRLs compared to HRLs, indicating performance inconsistency.

Limitations and Future Work

  • Dataset limitations: The study acknowledges that some datasets were small and offered limited support for LRLs, prompting a second experiment focusing on larger datasets. The FLORES-101 dataset was only evaluated on the first 100 sentences in a previous study, highlighting the need for more comprehensive evaluations.
  • Domain and genre impact: The influence of the domain and genre of training datasets on translation quality remains an open question for future work.
  • Language family hypothesis: While initial results hint at language families influencing performance, the current set of languages is too small to definitively confirm this hypothesis. Future work should explore this further.
  • Benchmarking framework: The authors advocate for the use of BENG to share evaluation data and improve comparability across systems and datasets, suggesting its continued development and adoption.

Summary updated

BibTeX
@inproceedings{silva2024benchmarking,
  author = {Ana Alexandra Morim da Silva and Nikit Srivastava and Tatiana Moteu Ngoli and Michael Röder and Diego Moussallem and Axel-Cyrille Ngonga Ngomo},
  title = {{Benchmarking Low-Resource Machine Translation Systems}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure