Learning Translation Quality Evaluation on Low Resource Languages from Large Language Models

Amirkeivan Mohtashami, Mauro Verzetti, Paul K. Rubenstein

Abstract

Evaluating machine translation (MT) quality for low-resource languages is challenging due to the scarcity of human-annotated data required for high-quality neural metrics like BLEURT. This paper proposes a knowledge distillation framework that leverages Large Language Models (LLMs) to generate synthetic training datasets. By creating sentence pairs and corresponding similarity scores via Chain-of-Thought prompting, we significantly enhance the performance of smaller, production-efficient evaluation models in languages where human data is limited.

Introduction

In machine translation, accurate evaluation is critical for system development. Traditional metrics such as BLEU rely on exact n-gram overlaps, often failing to capture semantic equivalence. While modern learned metrics like BLEURT address this by utilizing neural networks to understand semantics, their training depends on thousands of human-rated sentence pairs. For low-resource languages like Amharic, Urdu, or Mongolian, such data is prohibitively expensive or unavailable, leaving a gap in our ability to evaluate MT systems for the communities that need them most.

Methodology

We address this data scarcity through a four-step distillation pipeline using the PaLM LLM to train a smaller mT5-XL (3.7B parameter) student model:

  1. Sentence Selection: Seed sentences are extracted from the multilingual C4 (mC4) web-crawled dataset.
  2. Synthetic Pair Generation: PaLM is prompted to generate variations of seed sentences. We employ Chain-of-Thought (CoT) prompting, requiring the model to explain intended semantic changes before generating the text, which ensures higher adherence to constraints.
  3. Automated Scoring: PaLM rates the similarity of generated pairs on a 0–4 scale. We derive a continuous score by calculating a weighted average of probabilities assigned to each score token.
  4. Distillation: The synthetic data is combined with existing human-rated data to fine-tune the student model, creating a production-ready metric that inherits the LLM's multilingual capabilities.

Initial benchmarking reveals that while neural baselines and the PaLM teacher significantly outperform traditional metrics like SentenceBLEU, a performance gap remains in low-resource settings that synthetic data aims to fill.

Comparison of SentenceBLEU, the BLEURT-like baseline, and the PaLM teacher model across selected languages.

Key Results

The effectiveness of the synthetic data was evaluated using Pearson’s R correlation against human ratings on internal datasets. The results demonstrate substantial improvements across nearly all tested low-resource languages.

Pearson correlation results across various languages comparing the baseline to models trained on increasing amounts of synthetic data.

As detailed in the results table, increasing the volume of synthetic data (Small, Medium, Large) generally leads to higher correlation with human judgments. For example, in Amharic, correlation improved from a baseline of 0.273 to 0.300. In Urdu, it rose from 0.261 to 0.279. Notably, the learned models consistently outperformed SentenceBLEU, which exhibited near-zero correlation in several low-resource contexts (e.g., 0.025 for Mongolian vs. 0.156 for the distilled model).

Discussion and Insights

Our findings highlight several key aspects of LLM-based data generation:

  • Efficiency: Distillation allows for the creation of high-quality evaluation tools that do not require the massive infrastructure of a 540B-parameter LLM during inference.
  • Chain-of-Thought Utility: CoT prompting was essential for maintaining data quality; without it, the LLM often failed to follow complex generation constraints.
  • Cross-Lingual Transfer: We observed that adding data for one language often benefited others, suggesting the model learns a generalized representation of semantic similarity that transcends specific language pairs.

Limitations and Future Work

While effective, synthetic data can introduce noise if the LLM provides incorrect scores. Furthermore, distribution shifts between synthetic and real-world data can limit performance gains. Future research will explore Automatic Prompt Engineering (APE) to optimize data extraction and extend this pipeline to other tasks such as summarization and sentiment analysis in low-resource settings.

Summary updated

Abstract

Learned metrics such as BLEURT have in recent years become widely employed to evaluate the quality of machine translation systems. Training such metrics requires data which can be expensive and difficult to acquire, particularly for lower-resource languages. We show how knowledge can be distilled from Large Language Models (LLMs) to improve upon such learned metrics without requiring human annotators, by creating synthetic datasets which can be mixed into existing datasets, requiring only a corpus of text in the target language. We show that the performance of a BLEURT-like model on lower resource languages can be improved in this way.
BibTeX
@misc{mohtashami2023learning,
  author = {Amirkeivan Mohtashami and Mauro Verzetti and Paul K. Rubenstein},
  title = {{Learning Translation Quality Evaluation on Low Resource Languages from Large Language Models}},
  year = {2023},
  eprint = {2302.03491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.03491},
}
Figure