Abstract
Evaluating machine translation (MT) quality for low-resource languages is challenging due to the scarcity of human-annotated data required for high-quality neural metrics like BLEURT. This paper proposes a knowledge distillation framework that leverages Large Language Models (LLMs) to generate synthetic training datasets. By creating sentence pairs and corresponding similarity scores via Chain-of-Thought prompting, we significantly enhance the performance of smaller, production-efficient evaluation models in languages where human data is limited.
Introduction
In machine translation, accurate evaluation is critical for system development. Traditional metrics such as BLEU rely on exact n-gram overlaps, often failing to capture semantic equivalence. While modern learned metrics like BLEURT address this by utilizing neural networks to understand semantics, their training depends on thousands of human-rated sentence pairs. For low-resource languages like Amharic, Urdu, or Mongolian, such data is prohibitively expensive or unavailable, leaving a gap in our ability to evaluate MT systems for the communities that need them most.
Methodology
We address this data scarcity through a four-step distillation pipeline using the PaLM LLM to train a smaller mT5-XL (3.7B parameter) student model:
- Sentence Selection: Seed sentences are extracted from the multilingual C4 (mC4) web-crawled dataset.
- Synthetic Pair Generation: PaLM is prompted to generate variations of seed sentences. We employ Chain-of-Thought (CoT) prompting, requiring the model to explain intended semantic changes before generating the text, which ensures higher adherence to constraints.
- Automated Scoring: PaLM rates the similarity of generated pairs on a 0–4 scale. We derive a continuous score by calculating a weighted average of probabilities assigned to each score token.
- Distillation: The synthetic data is combined with existing human-rated data to fine-tune the student model, creating a production-ready metric that inherits the LLM's multilingual capabilities.
Initial benchmarking reveals that while neural baselines and the PaLM teacher significantly outperform traditional metrics like SentenceBLEU, a performance gap remains in low-resource settings that synthetic data aims to fill.
Key Results
The effectiveness of the synthetic data was evaluated using Pearson’s R correlation against human ratings on internal datasets. The results demonstrate substantial improvements across nearly all tested low-resource languages.
As detailed in the results table, increasing the volume of synthetic data (Small, Medium, Large) generally leads to higher correlation with human judgments. For example, in Amharic, correlation improved from a baseline of 0.273 to 0.300. In Urdu, it rose from 0.261 to 0.279. Notably, the learned models consistently outperformed SentenceBLEU, which exhibited near-zero correlation in several low-resource contexts (e.g., 0.025 for Mongolian vs. 0.156 for the distilled model).
Discussion and Insights
Our findings highlight several key aspects of LLM-based data generation:
- Efficiency: Distillation allows for the creation of high-quality evaluation tools that do not require the massive infrastructure of a 540B-parameter LLM during inference.
- Chain-of-Thought Utility: CoT prompting was essential for maintaining data quality; without it, the LLM often failed to follow complex generation constraints.
- Cross-Lingual Transfer: We observed that adding data for one language often benefited others, suggesting the model learns a generalized representation of semantic similarity that transcends specific language pairs.
Limitations and Future Work
While effective, synthetic data can introduce noise if the LLM provides incorrect scores. Furthermore, distribution shifts between synthetic and real-world data can limit performance gains. Future research will explore Automatic Prompt Engineering (APE) to optimize data extraction and extend this pipeline to other tasks such as summarization and sentiment analysis in low-resource settings.
Summary updated