Harnessing NLP for Indigenous Language Education: Fine-Tuning Large Language Models for Sentence Transformation

Mahshar Yahan, Dr. Mohammad IslamNAACL 2025

TL;DR

This paper explores fine-tuning large language models (LLMs) for sentence transformation tasks in Indigenous languages, specifically Bribri, Guarani, and Maya. By adapting LLMs with techniques like LoRA and few-shot prompting, the research demonstrates a promising approach to creating educational tools that can help preserve linguistic diversity.

What Problem Does This Paper Address?

Indigenous languages worldwide are facing endangerment due to various factors, including a lack of resources and limited integration into modern technologies. This poses a significant challenge for their preservation and transmission to future generations. Specifically, developing educational tools that can adapt and transform sentences based on grammatical rules is difficult for these low-resource languages. Traditional NLP approaches often struggle with the unique linguistic structures and limited data available for these languages. This paper aims to bridge this gap by leveraging the power of LLMs to create effective sentence transformation systems for Indigenous language education.

Methods

The researchers employed a multi-stage methodology to fine-tune LLMs for sentence transformation:

  • Data Preprocessing: The dataset was meticulously cleaned, with unnecessary entries (e.g., "NA" for no change) removed, and text standardized by removing punctuation and special characters. This step aimed to reduce noise and improve the model's ability to focus on meaningful linguistic patterns.
  • Tag Simplification: Complex grammatical instruction tags (e.g., "TYPE:NEG, TENSE:PRF_REC") were simplified into clear, actionable instructions (e.g., "Make the sentence negative and change to recent perfect tense"). This made the instructions more interpretable for the LLMs.
  • Prompt Design: Both zero-shot and few-shot prompting strategies were utilized.
  • Zero-shot prompts aimed to test the model's ability to perform transformations based solely on the instruction, without prior examples.
  • Few-shot prompts provided a few examples of source sentences, instructions, and target sentences to guide the model, which proved to be more effective.
  • Model Fine-tuning: Large Language Models (LLMs) like Llama 3.2 and XGLM 1.7B were fine-tuned using:
  • LoRA (Low-Rank Adaptation): This technique efficiently adapts LLMs by modifying only a subset of parameters, reducing computational cost and memory usage.
  • Bits-and-Bytes (BNB) Quantization: This method quantizes model weights to 4-bit, further reducing memory consumption during training.
  • Post-processing: A simple linear search was used to extract the "Target" sentence from the model's output, ensuring only the relevant transformed sentence was retained.

Datasets

The study utilized a dataset created for the AmericasNLP 2025 Shared Task 2. This dataset focuses on sentence transformation for Indigenous languages and includes:

  • Languages: Bribri, Guarani, Maya, and Nahuatl Omitlan.
  • Task: Sentence transformation based on linguistic instructions (tense, aspect, polarity, etc.).
  • Structure: The dataset is divided into Train, Development (Dev), and Test sets.
  • Characteristics: It contains 16 major categories with 68 unique values and features a mix of simple and complex instructions. The paper notes that the dataset size, particularly for training, is relatively small, presenting a challenge for robust model training.

Key Insights and Ideas

  • Few-shot prompting is crucial for low-resource languages: The research consistently showed that providing a few examples significantly boosts LLM performance compared to zero-shot approaches, especially for languages with limited data.
  • LLMs can be effectively adapted for grammatical transformations: Fine-tuning LLMs with techniques like LoRA and BNB quantization allows them to learn and apply complex grammatical changes accurately.
  • Llama 3.2 emerges as a strong performer: Among the tested models, Llama 3.2 demonstrated superior performance across all languages, particularly in few-shot settings.
  • Sentence transformation is a viable path for Indigenous language education: The success of this approach highlights the potential of NLP to create valuable educational tools that can aid in language preservation.

Key Results

The study evaluated models using BLEU and ChrF++ scores on development and test datasets.

  • Llama 3.2 (3B-Instruct) achieved the best performance:
    • For Bribri, it reached a BLEU score of 22.36 and ChrF++ of 50.46 (few-shot, dev set).
  • For Guarani, it achieved a BLEU score of 22.99 and ChrF++ of 58.30 (few-shot, dev set).
  • For Maya, it excelled with a BLEU score of 57.16 and ChrF++ of 82.48 (few-shot, dev set).
  • Few-shot prompting consistently outperformed zero-shot prompting across all models and languages.
  • XGLM 1.7B performed poorly in comparison.
  • On the test sets, Llama 3.2 (3B-Instruct) achieved competitive BLEU and ChrF++ scores, particularly for Maya, securing 9th place on the leaderboard.

Limitations and Future Work

Limitations:

  • Small Dataset Size: The limited amount of annotated data, especially for training, impacted model generalization and hindered language-specific adaptations (e.g., for Guarani).
  • Computational Constraints: These restricted broader experimentation with larger models or ensemble techniques.
  • Potential for Preprocessing Impact: The paper notes that while preprocessing is vital, the removal of apostrophes in some cases might have impacted results, as apostrophes can represent glottalization and differentiate minimal pairs in certain languages.
  • High Out-of-Vocabulary (OOV) Tokens: The tokenizer's rate of encountering unknown words could also be a challenge.

Future Work:

  • Enhance Dataset Quality and Diversity: Future research should focus on innovative data augmentation techniques like back-translation, contextual embedding-based augmentation, and syntax tree manipulation.
  • Ensemble Learning/Hybrid Modeling: Combining multiple models or approaches could further boost performance.
  • Neural Morphology Extensions: Integrating these extensions would help handle complex linguistic structures more effectively.
  • Expand to More Endangered Languages: Applying these methods to a wider range of endangered languages could significantly contribute to cultural heritage preservation through NLP.

Summary updated

BibTeX
@inproceedings{yahan2025harnessing,
  author = {Mahshar Yahan and Dr. Mohammad Islam},
  title = {{Harnessing NLP for Indigenous Language Education: Fine-Tuning Large Language Models for Sentence Transformation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure