The unreasonable effectiveness of large language models for low-resource clause-level morphology: In-context generalization or prior exposure?

Coleman HaleyNAACL 2024

TL;DR

This paper explores how well large language models (LLMs) can handle complex grammatical tasks in indigenous languages with very limited data. They found that LLMs, even with minimal examples (few-shot learning), perform surprisingly well, suggesting their strength lies in learning from the provided examples rather than relying on pre-existing knowledge of these specific languages.

What Problem Does This Paper Address?

Many of the world's languages are low-resource, meaning they have very little digital data available for NLP tasks. This makes it difficult to build effective NLP tools for them. This paper focuses on a specific, complex linguistic challenge: clause-level morphology, which involves how words change their form to convey grammatical meaning within a sentence. The authors investigate whether LLMs can tackle this challenge in indigenous languages with scarce data, and crucially, whether their success stems from general language understanding gained during pre-training or from their ability to learn from a few provided examples (in-context learning).

Methods

The researchers employed a simple few-shot prompting approach. They used three state-of-the-art LLMs: two closed-source models (GPT-3.5 and GPT-4) and one open-source model (Cohere's Command R+). The task was framed as a CSV completion problem, where the model was given input sentences and feature changes, and had to predict the resulting target sentence. To disentangle the effects of prior language exposure from in-context learning, they created a novel experiment: they permuted the alphabet of the languages. This preserved the grammatical structure of the task but removed any recognizable linguistic patterns, forcing the models to rely solely on the provided examples.

Datasets

The study utilized data from the AmericasNLP 2024 Shared Task on Creation of Educational Materials for Indigenous Languages. This task included three indigenous languages: Yucatec Maya (yua), Guaraní (grn), and Bribri (bzd). The training data for these languages was very limited, with at most 595 examples. The authors also created a permuted version of the development set for their specific experiment to test the in-context learning hypothesis.

Key Insights and Ideas

  • LLMs are surprisingly capable with minimal data: Even with very few examples, LLMs can achieve competitive performance on complex morphological tasks in low-resource languages.
  • In-context learning is the key driver: The study strongly suggests that LLMs' success is primarily due to their ability to learn from the few examples provided in the prompt, rather than leveraging extensive pre-existing knowledge of these specific indigenous languages.
  • Alphabet permutation is a powerful diagnostic tool: This novel technique effectively isolates the contribution of in-context learning by removing linguistic signal while preserving task structure.
  • Model choice matters: Despite Command R+ performing well on general benchmarks, it underperformed on Bribri and Guaraní compared to GPT-3.5 and GPT-4, highlighting that model architecture and pre-training can still influence performance on specific, low-resource tasks.
  • Subword tokenization can have subtle effects: The permuted alphabet experiment revealed that sometimes models perform better on the permuted data, suggesting that the way words are broken down into subword units can influence generalization.

Key Results

  • The best system, GPT-4, placed third overall in the shared task and achieved the highest performance for Yucatec Maya.
  • All three LLMs significantly outperformed the provided baseline across at least one language.
  • In the alphabet permutation experiment, systems generally performed similarly on both original and permuted datasets, reinforcing the dominance of in-context learning. For example, for Maya, all three models retained their performance on the permuted set.
  • While Bribri and Guaraní showed some performance degradation with alphabet permutation, the overall trend supported in-context learning as the primary factor.

Limitations and Future Work

The authors acknowledge that the results from the permuted alphabet experiment should be interpreted with caution due to run-to-run variability and resource constraints preventing extensive re-runs for confidence intervals. They also note that the reasons for performance differences between languages (Maya, Guaraní, Bribri) are not fully explained, though they mirror baseline performance, suggesting inherent task difficulty or data informativeness. Future work could delve deeper into characterizing the types of linguistic functions LLMs are better or worse at generalizing, and investigate the impact of data sparsity and specific linguistic phenomena (like morphophonological stem changes) on model performance.

Summary updated

BibTeX
@inproceedings{haley2024unreasonable,
  author = {Coleman Haley},
  title = {{The unreasonable effectiveness of large language models for low-resource clause-level morphology: In-context generalization or prior exposure?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure