Translating a low-resource language using GPT-3 and a human-readable dictionary

Micha Elsner, Jordan NeedleACL 2023

TL;DR

This paper explores using GPT-3 to translate words from Inuktitut, a low-resource polysynthetic language, by leveraging existing human-readable dictionaries. The authors demonstrate that GPT-3 can achieve reasonable translation quality by combining morpheme definitions, offering a promising avenue for low-resource language technology development without requiring large parallel corpora.

What Problem Does This Paper Address?

Developing natural language processing (NLP) tools for low-resource languages, especially polysynthetic ones like Inuktitut, is challenging due to the scarcity of parallel text data needed for training traditional machine translation (NMT) models. Simultaneously, these language communities often invest heavily in creating resources like dictionaries and grammar guides for revitalization and educational purposes. This paper addresses the gap of how to effectively utilize these existing, human-centric resources (dictionaries) to bootstrap NLP capabilities, specifically for translation, without the need for extensive parallel data. The goal is to bridge the divide between community-driven linguistic resources and the demands of machine translation.

Methods

The core of the methodology involves using GPT-3 (specifically the TEXT-DAVINCI-003 model) to translate Inuktitut words. The approach relies on a two-step process:

  1. Morphological Segmentation: The paper assumes access to a system that can provide canonical morphological segmentations of Inuktitut words (e.g., Uqailaut FST). This breaks down complex words into their constituent morphemes.
  2. Dictionary-Based Translation with GPT-3: For each morpheme in a segmented word, its definition is retrieved from a human-readable Inuktitut-English dictionary. GPT-3 is then prompted with these definitions to generate an English translation of the entire word.

Several prompting strategies were explored:

  • Concatenative Baseline: A simple baseline where morpheme definitions are just concatenated.
  • Definition Only: GPT-3 receives only the textual definitions of the morphemes.
  • Definition + Example: GPT-3 receives definitions and an example sentence showing the morpheme in use.
  • Multianswer: GPT-3 is prompted to provide multiple possible meanings for polysemous morphemes.
  • Multiprompt: Each combination of morpheme meanings is used to generate a separate prompt.
  • Grammar Lesson: Prompts are prefaced with grammatical information about Inuktitut.
  • Chain-of-Thought: A few-shot prompting method where GPT-3 explicitly identifies syntactic categories, subjects, objects, and modifier meanings before translating.

The paper also investigates the reverse task: using GPT-3 to infer morpheme definitions from whole-word translations.

Datasets

The primary dataset used is the Uqausiit dictionary (uqausiit.ca), an Inuktitut-English dictionary created by Inuit Uqausinginnik Taiguusiliuqtiit. This dictionary provides:

  • Morpheme definitions.
  • Example phrases with English translations.
  • Hand-annotated partial morphological decompositions.

The paper extracts data for two main tasks:

  • Translation: Single-word examples with multi-morphemic partial decompositions for development and testing.
  • Definition Prediction: Data for inferring morpheme definitions.

The paper notes that while Inuktitut has a relatively well-resourced parallel corpus (Nunavut Parliamentary Hansards), their approach aims to be applicable to even less-resourced polysynthetic languages.

Key Insights and Ideas

  • Leveraging Dictionaries for Low-Resource Translation: The most significant insight is that GPT-3 can effectively use human-readable dictionary definitions to translate complex words in a low-resource language, bypassing the need for large parallel corpora. This democratizes NLP for under-resourced languages by utilizing existing community knowledge.
  • GPT-3's Text-to-Text Capabilities: The paper highlights GPT-3's strength in text-to-text generation, enabling it to synthesize definitions into coherent translations, rather than just performing simple lookups.
  • Value of Examples and Definitions: Combining morpheme definitions with example usage significantly improves translation performance compared to definitions alone.
  • Handling Polysemy: GPT-3 can handle morphemes with multiple meanings reasonably well when prompted to provide multiple answers, suggesting that explicit enumeration of combinations (multiprompt) is not always necessary.
  • Grammar Information is Less Effective: Injecting explicit grammatical rules into prompts did not significantly improve translation performance, suggesting that LLMs might not yet effectively leverage abstract grammatical information for this task.
  • Definition Induction Challenges: The reverse task of inferring morpheme definitions from whole words is prone to errors, including hallucinations, making it less reliable than translation.

Key Results

  • Translation Performance: GPT-3 achieved BLEU scores of up to 18.5 for single-word translation using the "Definition+Example" and "Multianswer-Oracle" settings in a zero-shot scenario. Few-shot settings further improved scores, with "Definition+Example" reaching a BLEURT score of 0.52.
  • Baseline Comparison: The "Definition Only" approach performed poorly, with BLEU scores around 0.44, indicating GPT-3 has little prior knowledge of Inuktitut. The trivial "Concatenative" baseline scored slightly higher at 6.19 BLEU.
  • Definition Induction: The task of creating dictionary definitions using GPT-3 yielded much poorer results, with BLEURT scores around 0.33 for roots and 0.27 for functional morphemes. The system also exhibited a tendency to hallucinate definitions.
  • Qualitative Findings: While translations were not always perfect, many were considered "potentially useful for practical purposes." However, the "Definition Only" system often produced plausible but incorrect translations (confabulated meanings). The "Chain-of-Thought" method sometimes led to loss of information.

Limitations and Future Work

  • Low Statistical Power: The authors acknowledge that the numerical results have low statistical power due to the small test set size, meaning many differences are not statistically significant.
  • Reproducibility: Potential changes to OpenAI models could affect reproducibility.
  • Dependence on Segmentation: The method relies on an accurate canonical segmentation system, which might not be available for all low-resource languages.
  • Human Evaluation: The evaluations are entirely automated, and human evaluation by native Inuktitut speakers would improve reliability.
  • Definition Induction Errors: The definition induction task is prone to hallucinations and over-specificity, requiring careful filtering and potentially human curation.
  • Future Work:
    • Investigate better prompting strategies for grammar information.
  • Explore more sophisticated instruction-following models.
  • Test on better-resourced agglutinative languages to differentiate prompting techniques.
  • Integrate segmentation more closely with LLM prompting.
  • Explore the use of LLMs for more reliable dictionary definition generation.
  • Replicate results with smaller LLMs (e.g., T5).
  • Investigate how LLMs can exploit linguistic context to disambiguate translations.

Summary updated

BibTeX
@inproceedings{elsner2023translating,
  author = {Micha Elsner and Jordan Needle},
  title = {{Translating a low-resource language using GPT-3 and a human-readable dictionary}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure