TL;DR
This paper presents four systems developed by the "Meenzer Team" for the AmericasNLP 2024 shared task, focusing on automatically generating educational materials for low-resource Indigenous languages by applying specific grammatical transformations. Their best ensemble system achieved the highest performance on Guarani and secured second place overall, demonstrating the effectiveness of combining large language models with traditional sequence-to-sequence architectures.
What Problem Does This Paper Address?
The paper tackles the challenge of creating educational resources for endangered Indigenous languages, which often suffer from a lack of digital tools and data. Specifically, the AmericasNLP 2024 shared task (ST 2) aims to develop systems that can accurately apply grammatical modifications to source sentences in low-resource languages like Bribri, Guarani, and Maya. This is crucial for language learning, as it allows learners to practice understanding and replicating language structures through automated sentence transformations. The core difficulty lies in the low-resource nature of these languages, making it hard to train robust NLP models.
Methods
The Meenzer Team developed four ensemble systems, combining three main types of models:
- Character-level Pointer-Generator LSTMs: A traditional sequence-to-sequence architecture that can both generate characters from a vocabulary and copy them from the source sentence. This model was trained using a two-step approach: first on combined data across all languages, then fine-tuned on each target language.
- Mixtral 8x7B (Instruct) LLMs: A large language model fine-tuned using Supervised In-Context Learning (SICL). This approach integrates task instructions and examples directly into the training phase, differing from standard In-Context Learning (ICL) which doesn't update model parameters. They employed 4-bit quantization with LoRA adapters and experimented with varying numbers of examples per prompt.
- GPT-4: Another large language model used via In-Context Learning (ICL) with a significant number of examples.
The final systems were ensembles of these models, using majority voting for predictions, with the best-performing model on the development set breaking ties. They also explored different prompt ordering strategies for the LLMs to introduce diversity.
Datasets
The shared task utilized datasets for three Indigenous languages:
- Bribri
- Guarani
- Maya
For each language, training and development sets were provided. The test set's input was given for submission. The paper notes that the training data for Bribri and Guarani contained a significant number of unseen characters in the test set, posing a challenge.
Key Insights and Ideas
- Ensembling is Key: Combining different model architectures (LSTMs, Mixtral, GPT-4) proved highly effective, especially for the Guarani language, where the full ensemble achieved the best results.
- Mixtral's Strength: The Mixtral 8x7B model, particularly when used in an ensemble, showed strong performance, securing the highest accuracy on Maya and the second place overall.
- SICL vs. ICL: Supervised In-Context Learning (SICL) for fine-tuning Mixtral models outperformed standard In-Context Learning (ICL) for GPT-4, suggesting that parameter updates during training with examples are beneficial.
- Data Discrepancy: A significant gap between development and test sets, particularly in the proportion of unseen characters for Bribri and Guarani, highlights a major challenge in low-resource NLP and likely contributed to performance drops on the test set.
- Prompt Order Matters: For Mixtral, randomizing the order of in-context examples per epoch led to better performance than a fixed order, suggesting that exploring diverse example presentations can be beneficial.
Key Results
- Guarani: System 1 (LSTM + Mixtral + GPT-4 ensemble) achieved the highest performance on Guarani.
- Maya: System 4 (Mixtral ensemble) achieved the highest performance on Maya (tied with another team).
- Overall: System 4 (Mixtral ensemble) secured second place overall based on average accuracy.
- Development vs. Test Set: A notable discrepancy was observed, with overall performance dropping significantly from the development set to the test set. For instance, the Mixtral ensemble achieved 42.74% accuracy on the development set but 32.38% on the test set. The LSTM ensemble also saw a substantial drop.
- Ablation Study: Ensembling Mixtral models boosted accuracy by ~4.19 points compared to a single Mixtral model. SICL for Mixtral achieved 36.19% accuracy, significantly outperforming ICL for GPT-4 (20.55%). Randomizing prompt order improved performance by ~3.43% on average.
Limitations and Future Work
- Data Scarcity and Out-of-Vocabulary Issues: The significant presence of unseen characters in the test sets, especially for Bribri and Guarani, is a major limitation. This suggests that future work should focus on robust handling of unknown tokens, potentially through character-level processing or more advanced sub-word tokenization strategies.
- LSTM Performance Decline: The sharp drop in performance for LSTM-based systems on the test set warrants further investigation. The authors suggest that the character-level nature of LSTMs might make them more susceptible to unknown characters encountered during training.
- Model Diversity: While ensembles were effective, exploring more diverse model combinations or advanced ensembling techniques could further improve performance.
- Generalizability: The success of SICL for Mixtral suggests that further research into effective in-context learning and fine-tuning strategies for low-resource languages is a promising avenue.
Summary updated