Learning-From-Mistakes Prompting for Indigenous Language Translation

You Cheng Liao, Chen-Jui Yu, Chi-Yi Lin, He-Feng Yun, Yen-Hsiang Wang, Hsiao-Min Li, Yao-Chung FanACL 2024

TL;DR

This paper introduces a novel "Learning-from-Mistakes" (LFM) prompting strategy to significantly improve the translation capabilities of Large Language Models (LLMs) for extremely low-resourced indigenous languages. By leveraging existing LLM strengths, limited parallel data, and a word-level dictionary, the authors demonstrate that even languages unseen during LLM pre-training can be translated with surprising accuracy.

What Problem Does This Paper Address?

Translating between well-resourced languages like English and Chinese is a mature field for LLMs. However, a significant gap exists in their ability to handle indigenous languages, which often have limited digital data, complex grammars, and unique cultural nuances. Existing LLM approaches typically rely on languages that were part of their extensive pre-training. This paper tackles the challenge of translating into indigenous languages that LLMs have likely never encountered before, aiming to make LLMs more versatile tools for language preservation and revitalization.

Methods

The authors propose a cumulative, layered approach to prompting LLMs:

  • KNN-Prompting with Retrieved Prompting Context (RPC): This foundational step involves finding contextually similar translation examples (k-nearest neighbors) and individual word translations from a dictionary. The LLM uses this context to learn grammatical norms and sentence structures.
  • Chain-of-Thought (CoT) Prompting: Building on RPC, this method guides the LLM to explicitly show its reasoning process. It uses CoT demonstrations to illustrate how to leverage the RPC to construct accurate translations, enhancing the LLM's ability to follow instructions and utilize the provided context.
  • Learning-from-Mistakes (LFM) Prompting: This is the core innovation. It's a two-stage process:
  • Trial Translation: The LLM performs an initial translation using CoT KNN-Prompting.
  • Error Correction: The LLM is then presented with its own trial translation alongside the correct translation and examples of past errors. It's tasked with refining its initial translation based on this feedback, effectively learning from its mistakes to improve accuracy and linguistic precision.

Datasets

The primary dataset used for evaluation comes from the 'Klokah' website, provided by the Foundation for the Research and Development of Indigenous Languages in Taiwan. This includes:

  • Indigenous Language Corpora: For each language, a dataset of 450 sentences with corresponding Chinese translations.
  • Word-Level Dictionaries: Dictionaries containing 1000 words with single-word translations between Chinese and the indigenous language.

The data was split into test (100 sentences) and reference (350 sentences) sets for evaluation. The paper also mentions the Amis-Mandarin dataset (5,751 sentences) as a relevant resource for indigenous language translation research.

Key Insights and Ideas

  • LLMs as "Language Compilers": The paper frames LLMs not just as translators but as "compilers" that can internalize syntactic structures from limited data.
  • The Power of Iterative Refinement: The LFM strategy highlights that explicitly feeding LLMs their own errors and correct examples is crucial for improving performance, especially in low-resource settings.
  • Bridging the "Unseen Language" Gap: The core contribution is demonstrating that LLMs can be effectively prompted to translate languages they were not pre-trained on, opening up possibilities for many under-represented languages.
  • CoT Enhances RPC: Chain-of-Thought prompting significantly boosts the effectiveness of the retrieved context, showing that guiding the LLM's reasoning process is key.

Key Results

  • Significant BLEU Score Improvements: The LFM Prompting method consistently outperformed baseline methods (Zero-shot, 20-shots, KNN-Prompting, CoT Prompting) across multiple indigenous languages (Southern Amis, Coastal Amis, Wanda Tayal, Siji Tayal, Duda Seediq). For instance, in Southern Amis, LFM achieved a BLEU3 score of 8.2, a substantial increase from Zero-shot (0.0) and even surpassing CoT Prompting (5.9).
  • Expert Validation: A qualitative review by a Coastal Amis language expert confirmed the superiority of LFM, noting its contribution to linguistic precision and affirming its role in enhancing translation quality. The expert also identified specific areas for improvement, such as handling time adverbs and sentence structure, which LFM is expected to address.

Limitations and Future Work

  • Ambiguity in Low-Resource Languages: For languages like Southern Amis, determining subtle distinctions between synonyms (e.g., different terms for "we") requires native speaker expertise, which is difficult to capture solely through data.
  • BLEU Metric Limitations: The BLEU score provides a single, standardized answer, which may not always align with the true contextual meaning or fluency.
  • LFM's Reliance on Prior Examples: The LFM method's corrections are guided by prior examples; the LLM itself doesn't possess independent reasoning for structural adjustments.
  • Insufficient Few-Shot Data: While the methods show data-driven enhancements, they don't fully overcome the fundamental challenge of insufficient few-shot data, which can lead to inconsistent outcomes.

Future work should focus on addressing these limitations, particularly by developing more robust methods for handling linguistic ambiguity and improving the LLM's independent reasoning capabilities in translation.

Summary updated

BibTeX
@inproceedings{liao2024learning,
  author = {You Cheng Liao and Chen-Jui Yu and Chi-Yi Lin and He-Feng Yun and Yen-Hsiang Wang and Hsiao-Min Li and Yao-Chung Fan},
  title = {{Learning-From-Mistakes Prompting for Indigenous Language Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure