TL;DR
This paper investigates the performance of state-of-the-art Large Language Models (LLMs) like GPT-4 for machine translation of Spanish into 11 indigenous languages of South America. Despite LLMs' general capabilities, they perform poorly on these extremely low-resource languages, often producing boilerplate text and inaccurate translations. Fine-tuning existing multilingual models like M2M100, especially when augmented with k-nearest-neighbor techniques, offers significantly better results.
What Problem Does This Paper Address?
Machine translation (MT) has made remarkable strides, particularly with the advent of large language models (LLMs). However, these advancements have largely focused on high-resource languages. Extremely low-resource languages, often indigenous languages with limited digital data, remain a significant challenge for MT systems. The authors aim to understand how well current LLMs, specifically GPT-4, perform on these languages and to explore alternative approaches that can improve translation quality in such data-scarce environments. This is crucial for preserving linguistic diversity and enabling communication for communities with limited access to translation technologies.
Methods
The researchers evaluated four different approaches:
- GPT-4 API: They used the GPT-4 API to translate Spanish into the target indigenous languages, employing a specific prompt designed for machine translation.
- Bilingual Transformer Models: They trained standard Transformer models using parameters optimized for extremely low-resource data regimes, leveraging the Fairseq library.
- Fine-tuned M2M100: They fine-tuned the multilingual M2M100 model for each of the 11 target languages. Since these languages are not directly supported by M2M100, they used a Swahili language tag as a proxy, based on prior work showing this doesn't significantly impact quality.
- Fine-tuned M2M100 + kNN-MT: This approach augmented the fine-tuned M2M100 model with a k-nearest-neighbor (kNN) machine translation component. The kNN-MT system uses a datastore of cached examples to retrieve relevant translations, which are then combined with the NMT output. This method was developed to avoid directly encoding the development set into the model weights, aiming for a more robust approach.
Datasets
The primary dataset used was the parallel data provided by the AmericasNLP 2023 Shared Task organizers for training across all South American languages. For Chatino, they also utilized the Chatino Speech Corpus Archive Dataset. The development set was used for hyperparameter tuning and for creating the kNN-MT datastores.
Key Insights and Ideas
- LLMs are not a silver bullet for low-resource languages: Despite GPT-4's impressive general capabilities, it struggles significantly with extremely low-resource indigenous languages, producing translations that are often inaccurate and contain unwanted boilerplate text.
- Fine-tuning multilingual models is key: Fine-tuning pre-trained multilingual models like M2M100 on the specific low-resource language pairs yields substantially better results than using general-purpose LLMs or training bilingual models from scratch.
- kNN-MT offers a powerful boost: Augmenting fine-tuned multilingual models with k-nearest-neighbor retrieval can further enhance translation quality, especially in low-resource settings. This approach allows for leveraging existing data more effectively without direct overfitting.
- Boilerplate text is a common LLM issue: GPT-4 frequently added extraneous phrases like "Feel free to make adjustments..." to its translations, which required post-processing.
Key Results
The paper reports Chrf++ scores for various model combinations across 11 language directions.
- GPT-4: Performed better than bilingual models for 7 out of 10 directions on both development and test sets, but overall scores were very low.
- Bilingual Models: Achieved very low scores, indicating they are not effective translators for these languages.
- Fine-tuned M2M100: Showed significantly better performance than GPT-4 and bilingual models.
- Fine-tuned M2M100 + kNN-MT: This system achieved the best results for 10 out of 11 language directions and was the top-performing model that did not encode the development set. It ranked first for Spanish-Bribri, Spanish-Asháninka, and Spanish-Nahuatl.
Limitations and Future Work
The authors acknowledge that their kNN-MT approach, while effective, operates in a "grey area" regarding competition rules by using the development set for datastores. This led them to withdraw this model from the competition track due to concerns about fairness. They suggest that future work could focus on developing more robust methods for leveraging available data without violating strict competition guidelines, and further exploring the potential of fine-tuned multilingual models for indigenous language translation.
Summary updated