Abstract
This study evaluates state-of-the-art multilingual large language models (mLLMs) on English-Amharic translation, a historically underserved low-resource language pair. By integrating automatic metrics with a rigorous human error analysis using the Multidimensional Quality Metrics (MQM) framework, the research identifies critical failure modes. While Meta’s NLLB-3.3B model demonstrates superior performance, significant challenges remain, particularly regarding "hallucinated" mistranslations and the handling of named entities.
Introduction
Machine Translation (MT) has achieved remarkable success in high-resource languages, yet low-resource languages like Amharic—spoken by over 20 million people in Ethiopia—continue to lag. The primary issue is not just lower translation quality but a lack of understanding regarding the specific failure patterns of massive multilingual models in these contexts. This paper addresses this gap through a systematic bidirectional error analysis of English-Amharic translation.
Methodology
The researchers evaluated two major families of mLLMs: NLLB-200 (ranging from 600M to 3.3B parameters) and M2M (418M and 1.2B parameters). The evaluation pipeline combined standard automatic metrics with a deep-dive human assessment.
The human evaluation involved three proficient annotators who used the Multidimensional Quality Metrics (MQM) framework to categorize errors by type (e.g., Mistranslation, Addition, Omission) and severity (Minor, Major, Critical). This approach allowed for a more nuanced understanding than automatic scores alone.
Datasets
The study utilized the Lesan AI dataset, a parallel corpus of 987 Amharic and 1,915 English sentences. This dataset provides a robust benchmark, featuring diverse content from news, Wikipedia, Twitter, and conversational text, making it representative of real-world language use.
Key Insights
The Named Entity Challenge
A significant finding was the models' tendency to fail on proper nouns. For example, the M2M model translated "Dr. Abiy" (Ethiopia’s Prime Minister) as "Al-Qaeda," and "Gojam" (an Ethiopian region) as "Gujarat." This suggests models often default to high-resource entities when encountering unfamiliar low-resource terms.
Literalism and Contextual Failures
Models frequently struggled with polysemy, often defaulting to literal translations. For instance, the Amharic word for "Writer" was incorrectly translated as "Secretary" due to a lack of contextual grounding. This highlights the need for models to better capture semantic nuances in low-resource settings.
Directional Disparity
Performance was generally higher for English-to-Amharic translation than vice versa. This disparity likely stems from the models' extensive exposure to English during pre-training, which enhances their understanding of the source text compared to Amharic.
Results and Error Analysis
NLLB-3.3B emerged as the top performer, achieving a BLEU score of 26.7 for Amharic-to-English and 22.0 for English-to-Amharic. However, human quality scores revealed a stark contrast: 84.77 for English-to-Amharic versus only 52.47 for Amharic-to-English.
As shown in the error distribution, Mistranslation was the most prevalent error type by a wide margin. Interestingly, the models maintained high "fluency" in terms of spelling and punctuation even when the semantic meaning was entirely lost or "hallucinated."
Conclusion and Future Work
While NLLB-3.3B represents a significant step forward, the study highlights critical areas for improvement. Future research should focus on integrating Named Entity Recognition (NER) to prevent identity-swap errors and developing larger, more diverse Amharic datasets to improve model grounding in low-resource contexts. Addressing these semantic and entity-level failures is crucial for making MT truly inclusive for all languages.
Summary updated