TL;DR
This paper explores using Large Language Models (LLMs) to improve machine translation for low-resource indigenous languages by providing them with grammar reference materials. The approach shows promise, particularly for translating into Spanish, suggesting LLMs can leverage linguistic knowledge to handle languages they haven't been extensively trained on.
What Problem Does This Paper Address?
Traditional machine translation (MT) systems thrive on massive parallel corpora (pairs of sentences in two languages). However, for many indigenous languages, such extensive data simply doesn't exist, creating a significant "low-resource" challenge. This paper tackles this data scarcity by investigating whether LLMs, when "taught" with grammar books and dictionaries, can effectively post-correct translations generated by a more conventional MT system. The goal is to bridge the gap for languages that are severely underrepresented in digital translation resources.
Methods
The core of this research is a prompt-based approach for LLMs. Instead of relying solely on the LLM's pre-existing knowledge, the authors construct detailed prompts that include:
- Prefix: A clear instruction defining the translation task, source, and target languages.
- Grammar Book: The full text of a grammar document for the target indigenous language.
- Dictionaries: Bilingual dictionary entries relevant to the source sentence, retrieved using longest common subsequence (LCS) matching.
- Parallel Sentences: A few example sentence pairs to provide in-context learning.
- Suggested Translation: A preliminary translation generated by a finetuned NLLB (No Language Left Behind) model.
- Suffix: A final instruction reiterating the task and encouraging the LLM to provide a translation even if it's not perfectly fluent.
The authors use the GPT-4o-mini LLM, leveraging its large context window to accommodate the grammar materials. They also train separate NLLB 3.3B models for each translation direction (indigenous language to Spanish, and Spanish to indigenous language) to serve as the "suggested translation" input.
Datasets
The primary datasets used are from the AmericasNLP 2025 Shared Task on Machine Translation into Indigenous Languages. This includes:
- Parallel Sentences: The training data provided for the shared task.
- Dictionaries: Obtained from Pan-Lex. Dictionaries with fewer than 100 words were excluded.
- Grammar Books: Digitized grammar documents sourced from the DReaM corpus. These underwent minor manual cleanup.
The dev and test datasets for the 13 Latin American indigenous languages participating in the AmericasNLP 2025 task were used for evaluation.
Key Insights and Ideas
- LLMs as Linguistic Learners: The most significant insight is that LLMs can indeed leverage grammar materials to improve translation quality, even for languages they haven't seen extensively during pretraining. This demonstrates their potential as adaptable learners in low-resource scenarios.
- Directionality Matters: The LLM-based post-correction approach is more effective for translating into Spanish (xx→es) than for translating from Spanish (es→xx). This suggests that while LLMs can use grammar to understand an unfamiliar language better, they still struggle to produce fluent output in it without more direct exposure.
- Hybrid Approach Synergy: Combining a dedicated MT system's output with LLM-based grammar-informed correction offers a promising hybrid strategy. The LLM acts as a sophisticated editor, refining the initial translation.
- Grammar Book Quality is Crucial: The effectiveness of the grammar books is highlighted, but also their limitations. Digitized versions can suffer from scanning artifacts, and multi-column text or tables can lose information, potentially hindering the LLM's ability to fully utilize them.
Key Results
The paper reports chrF++ scores on the dev and test datasets.
- XX→ES (Indigenous Language to Spanish): The LLM-based system consistently outperformed the NLLB baseline, showing improvements in chrF++ scores across many languages. For example, on the test set, the GPT system achieved higher chrF++ scores than the NLLB baseline for most languages in Table 1.
- ES→XX (Spanish to Indigenous Language): The finetuned NLLB system generally performed better than the LLM-based approach in this direction. This reinforces the idea that LLMs are better at understanding and correcting into a known language (Spanish) than producing output in an unfamiliar one.
Limitations and Future Work
- Grammar Book Availability and Quality: Obtaining high-quality, digitized grammar books for many languages is a significant challenge. The DReaM corpus helps, but imperfections in digitization (scanning artifacts, lost formatting) limit the LLM's ability to fully leverage this information.
- Closed Models: The use of closed-source LLMs (like GPT-4o-mini) means researchers don't have access to the model's architecture, weights, or training data, making it harder to understand why certain improvements occur or to replicate the exact setup.
- Prompt Engineering: The authors acknowledge that the current prompt format and generation paradigm might not be optimal. Future work could explore different prompt structures, more sophisticated retrieval mechanisms, or techniques like Retrieval-Augmented Generation (RAG) to improve efficiency and effectiveness.
- Common Language Influence: The paper notes that the common language for parallel sentences was Spanish, while prompts and some grammar books were in English. The interaction and effect of having these multiple languages in the prompt is an area for further investigation.
- Ablation Studies: While not performed for all languages due to time constraints, the authors suggest future work could include ablation studies to precisely quantify the contribution of each component (dictionaries, parallel sentences, grammar books) to the overall performance.
Summary updated