TL;DR
This paper tackles the challenge of evaluating machine translation for low-resource Indigenous languages by fine-tuning a powerful multilingual language model (LaBSE) to better understand their unique linguistic structures. By integrating these fine-tuned embeddings into existing metrics like YiSi-1 and COMET, the authors achieve improved correlation with human judgments, offering a more reliable way to assess translation quality for these underrepresented languages.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) is crucial for development, but traditional metrics like BLEU struggle with the complexities of Indigenous languages. These languages often feature rich morphology, non-standardized spelling, and unique grammatical structures that differ significantly from the well-resourced languages for which metrics like BLEU were designed. This makes it difficult to accurately assess MT performance, hindering progress in making Indigenous language content more accessible. While newer, embedding-based metrics like COMET and BERTScore show promise, they typically require vast amounts of training data, which is scarce for Indigenous languages. This paper aims to bridge this gap by adapting and fine-tuning existing powerful multilingual models to better capture the semantic nuances of Indigenous languages, thereby improving MT evaluation for them.
Methods
The core of this work lies in fine-tuning the Language-agnostic BERT Sentence Encoder (LaBSE). LaBSE is a multilingual model trained to produce sentence embeddings where translations in different languages are mapped to similar points in a shared semantic space. The authors fine-tune LaBSE using a contrastive learning approach on parallel data from Indigenous languages (Guarani, Bribri, Nahuatl) and Spanish. Crucially, they focus on aligning the non-Spanish sentences to preserve the pre-trained multilingual knowledge.
These fine-tuned LaBSE embeddings are then integrated into two established MT evaluation metrics:
- YiSi-1: A semantic similarity metric that compares hypothesis and reference translations. The authors explore using embeddings from different layers of the fine-tuned LaBSE, investigating both language-specific best layers and an average across languages.
- COMET-DA: A regression-based MT evaluation model. The fine-tuned LaBSE embeddings serve as the backbone for COMET, and the authors experiment with Mean Absolute Error (MAE) and Mean Squared Error (MSE) loss functions during training.
Datasets
The primary datasets used are:
- AmericasNLP MT Shared Task parallel data: This dataset covers 13 Indigenous languages.
- Nahuatl corpus: An additional corpus for Nahuatl was used to address data scarcity for this language.
- Human evaluation data (DA scores): Used for training and evaluating the MT metrics.
The paper does not introduce a new benchmark dataset but rather leverages existing ones for the AmericasNLP Shared Task.
Key Insights and Ideas
- Fine-tuning LaBSE is effective for Indigenous languages: Even with limited data, fine-tuning LaBSE significantly enhances its ability to capture the semantic representations of Guarani, Bribri, and Nahuatl. This suggests that pre-trained multilingual models can be successfully adapted to low-resource languages.
- Layer averaging matters for YiSi-1: Averaging embeddings from specific layers (layers 4-6) in the fine-tuned LaBSE improved YiSi-1's performance, indicating that different layers capture distinct semantic information, and a combination can be more robust.
- Aligning mean-pooled tokens is better than CLS token: The authors found that aligning mean-pooled token embeddings was more effective than aligning only the [CLS] token for fine-tuning, suggesting that a more comprehensive update of token representations is beneficial.
- Language-specific vs. cross-language averaging: While language-specific layer selection for YiSi-1 showed promise, cross-language averaging (layers 4-6) provided the best overall performance, highlighting the benefit of a more generalized approach when data is limited.
Key Results
The paper reports the following key findings:
- YiSi-1 with layer-averaged embeddings (layers 4-6) achieved the highest average correlation across the three languages among the submitted systems.
- COMET-based metrics demonstrated competitive performance for Guarani, with the MSE loss variant achieving the highest Pearson correlation for this language.
- For Bribri, YiSi-1 with layer averaging had the highest Spearman correlation.
- Nahuatl proved to be particularly challenging, with none of the submitted metrics surpassing the ChrF++ baseline for Spearman correlation. However, layer-averaged YiSi-1 still scored the highest among their systems for both Spearman and Pearson correlations.
- On average, the layer-averaged YiSi-1 metric performed best across all languages.
Limitations and Future Work
- Nahuatl remains challenging: The paper acknowledges that Nahuatl was particularly difficult to evaluate, and their metrics did not surpass the ChrF++ baseline for Spearman correlation.
- Limited data: The scarcity of data for Indigenous languages is a persistent challenge.
- Future Work: The authors suggest training COMET with more data to further improve its performance. They also propose investigating language-specific adjustments to metrics to better handle challenging languages like Nahuatl. Additionally, exploring other embedding models and fine-tuning strategies could yield further improvements.
Summary updated