The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English

Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Miguel Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, Marc'Aurelio RanzatoEMNLP 2019 · OpenReview

Abstract: Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, Marc’Aurelio Ranzato. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

While Neural Machine Translation (NMT) has reached near-human performance for major language pairs like English-Chinese, the vast majority of the world’s languages remain "low-resource." These languages lack the massive parallel corpora—millions of translated sentence pairs—required to train modern models.

A critical bottleneck in this field is evaluation. Without high-quality, publicly available benchmarks, researchers cannot accurately track progress. Existing datasets for low-resource languages are often small, noisy, or restricted to narrow domains like religious texts. This paper introduces the FLORES benchmark, specifically designed to test how well MT systems handle diverse, "out-of-domain" Wikipedia content in Nepali and Sinhala—two languages with vastly different syntax and morphology compared to English.

Methodology

To ensure the benchmark's quality, the authors developed a rigorous pipeline:

  1. Document Selection: 2,500 sentences were selected from Wikipedia across diverse topics (history, science, etc.) in English, Nepali, and Sinhala.
  2. Professional Translation: Each sentence was translated twice by different professional translators to ensure accuracy.
  3. Automatic Filtering: N-gram language models and "char-BLEU" metrics were used to identify and remove translations that were too similar (copying) or too divergent (errors).
  4. Human Quality Control: A "direct assessment" method was employed where three raters scored translations on a 0–100 scale. Only those scoring above 70 were retained. Fluency was further verified using Amazon Mechanical Turk.

Datasets

The FLORES benchmark provides evaluation sets (dev, devtest, and test) for Nepali–English and Sinhala–English, totaling approximately 3,000 sentence pairs per language. For training, the authors utilized a variety of existing resources, ranging from religious texts to movie subtitles, as well as large-scale monolingual data for semi-supervised learning.

Summary of training data available for Nepali-English and Sinhala-English.

Key Insights and Results

The Failure of Unsupervised MT

While unsupervised models (learning from non-parallel text) show promise for high-resource pairs, they failed completely on these distant language pairs, yielding BLEU scores near zero. This failure is likely due to poor word embedding initialization when languages do not share an alphabet or structural similarities.

The "Translationese" Effect

The authors observed that sentences originally written in the native language (e.g., Nepali) are significantly harder for models to translate into English than sentences that were originally English and then translated into the native language. This suggests that "translationese"—the subtle stylistic markers of translated text—makes the task artificially easier for models.

Average BLEU scores for Nepali-English by document ID, showing the performance gap between English-original and Nepali-original content.

Quantitative Performance

  • Low Baselines: Even the best supervised NMT models achieved BLEU scores below 8.0 for both pairs, a stark contrast to the 30s or 40s seen in high-resource settings.
  • Back-translation: Semi-supervised learning via back-translation (creating synthetic training data from monolingual text) proved most effective, nearly doubling BLEU scores.
  • Multilingual Boost: For Nepali, incorporating Hindi (a related language) into training improved scores by 6.5 points, highlighting the value of transfer learning.
  • Filtering: Using LASER to filter noisy web-scraped data (Paracrawl) improved quality by over 5 BLEU points.

Conclusion and Future Work

The primary challenge remains domain drift; models trained on entertainment or religious data struggle with the encyclopedic nature of the FLORES set. Future research should focus on improving unsupervised initialization for distant language pairs and developing more robust domain adaptation techniques. By providing a reliable evaluation standard, FLORES aims to democratize MT research for the world's underserved languages.

Summary updated

BibTeX
@inproceedings{guzman2019flores,
  author = {Francisco Guzmán and Peng-Jen Chen and Myle Ott and Juan Miguel Pino and Guillaume Lample and Philipp Koehn and Vishrav Chaudhary and Marc'Aurelio Ranzato},
  title = {{The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=aTXp_7v3GI_},
}
Figure