How Good is NLLB for Low-resource Languages? A Study on the Genoese Language

Davide Buscaldi, Paolo RossoCLiC-it 2023 · OpenReview

Abstract

While Meta’s NLLB-200 model aims to provide machine translation for 200 languages, its efficacy on extremely low-resource, non-standardized languages remains under-explored. This study evaluates NLLB-200 on Genoese, a variant of the endangered Ligurian language. Our findings indicate that while the model represents a significant technical milestone, it suffers from severe limitations in handling local context, named entities, and orthographic variations, often resulting in hallucinations or cross-lingual interference from dominant sister languages.

Introduction

Machine translation (MT) has historically focused on high-resource languages, leaving those with scarce digital footprints behind. Meta AI’s No Language Left Behind (NLLB-200) project seeks to bridge this gap by supporting 200 languages, including Ligurian. However, Ligurian presents unique challenges: it has a minimal web presence (approximately 11,000 short Wikipedia articles) and lacks standardized syntax or spelling due to its strong oral tradition.

Spelling variations in Genoese illustrate the lack of standardization.

As shown in the figure above, a single sentence can have multiple orthographic representations depending on the local variant. This study investigates whether a massive multilingual model trained primarily on translated Wikipedia articles can handle such nuances and high variance in the real world.

Methodology

The authors evaluated four NLLB-200 variants to understand the impact of scale and distillation:

  1. Standard models: 1.3B and 3.3B parameters.
  2. Distilled models: 600M and 1.3B parameters.

Zero-shot translations were performed in both directions (Italian ↔ Genoese). Evaluation metrics included spBLEU, chrF, and Translation Error Rate (TER). Beyond quantitative metrics, the authors conducted a qualitative error analysis focusing on tokenization, language-family embeddings (LASER-3), and the handling of local toponyms.

Datasets

The NLLB-200 training data for Ligurian (NLLB-Seed) consisted of 6,193 sentences sampled from English Wikipedia and translated via Italian. To provide a more realistic evaluation, the authors created a new parallel corpus of 283 sentences sourced from local culture:

  • De André + Trilli: 95 lines from famous Fabrizio De André songs (e.g., "Crêuza de mä").
  • Dickens: 188 sentences from a Genoese translation of Charles Dickens' Pictures of Italy.

Key Results and Discussion

The study found a significant disparity in performance based on translation direction. Translation from Genoese to Italian was notably more successful than the reverse across all model sizes.

Performance metrics for Italian to Genoese translation showing low spBLEU and high TER.

As detailed in the table above, the Italian to Genoese direction yielded poor results, with the 3.3B model achieving only 5.9 spBLEU on the full dataset. Furthermore, the TER scores frequently exceeded 100, indicating massive overgeneration. This was particularly prevalent in distilled models, which often fell into infinite repetition loops (e.g., repeating the tokens "pe pe pe..."). The authors suspect that word-level distillation struggles when training data is sparse, skewing probability distributions toward a few frequent subwords.

Qualitative Analysis

  • Contextual Blindness: Because the training data was derived from general Wikipedia articles, the model lacks the cultural and geographic context of the Liguria region.
  • Toponym Failures: The model failed to correctly translate "Genova" to the Genoese Zena, often defaulting to the English "Genoa" or leaving it in Italian.
  • Cross-lingual Interference: Due to the reliance on LASER-3 embeddings, which group languages by family, the model occasionally substituted Genoese words with related Romance languages. For example, "San Pietro" (Saint Peter) was translated as "San Peixe"—which is Portuguese for fish.

Conclusion

While NLLB-200 is a significant advancement, it is not yet a robust solution for endangered languages like Genoese. The authors suggest that future improvements should focus on fine-tuning with local, culturally relevant datasets rather than generic Wikipedia translations. Additionally, integrating external bilingual lexicons or using bilingual lexicon induction could help the model better handle Out-Of-Vocabulary (OOV) named entities and local toponyms.

Summary updated

BibTeX
@inproceedings{buscaldi2023how,
  author = {Davide Buscaldi and Paolo Rosso},
  title = {{How Good is NLLB for Low-resource Languages? A Study on the Genoese Language}},
  booktitle = {CLiC-it},
  year = {2023},
  url = {https://openreview.net/forum?id=v40Ef58HEf},
}
Figure