TL;DR
This paper tackles the challenge of adapting Large Language Models (LLMs) to understand and generate text in under-resourced South Slavic micro-dialects. By employing data augmentation techniques, the authors significantly improve LLM performance on dialectal commonsense reasoning tasks, demonstrating the practical utility of these methods for low-resource language scenarios.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable capabilities in understanding and generating text in well-resourced, standard languages. However, their performance on non-standard dialectal varieties, especially those with limited digital resources, remains an open question. The Dialect-Copa shared task specifically aims to evaluate LLMs' commonsense reasoning abilities on South Slavic micro-dialects, which present unique challenges due to their linguistic variations and scarcity of training data. This paper addresses this gap by proposing and evaluating data augmentation strategies to enhance LLM performance on these challenging dialectal tasks.
Methods
The authors' approach centers on data augmentation to overcome the limited size of the Dialect-Copa training dataset (only 400 instances per language). They explore several augmentation techniques:
- Transliteration: Converting Macedonian (mk) data from Cyrillic to Latin script to align with the Latin script prevalent in other datasets.
- Reverse-Augmentation: Swapping the premise and correct answer choice in existing examples to create new training instances, effectively doubling the dataset size.
- Synthetic Data Generation:
- Using GPT-4 to generate synthetic English COPA-style data.
- Translating this synthetic English data into target dialects (Croatian, Macedonian, Serbian, Slovenian) using the NLLB-6B model.
- Leveraging the Claude-3 language model with custom conversion rules and examples to generate synthetic Chakavian data, addressing the complete lack of training data for this dialect.
They then combine these augmented datasets with the original training data in various configurations (detailed in Table 4) to fine-tune and instruction-tune selected language models. For smaller, language-family-focused models like BERTić, they use full fine-tuning. For larger models like Aya-101, they employ LoRA (Low-Rank Adaptation) to reduce computational costs while maintaining performance.
Datasets
- Dialect-Copa Dataset: The primary dataset used for training, validation, and testing. It covers 8 languages and dialects, with a focus on South Slavic micro-dialects. The test set includes Chakavian, Cerkno, and Torlak dialects.
- English COPA-style synthetic dataset: Generated by GPT-4 and used as a source for translation into target dialects.
Key Insights and Ideas
- Data Augmentation is Crucial: The study strongly suggests that data augmentation is a highly effective strategy for improving LLM performance on low-resource dialectal tasks, especially when dealing with limited original training data.
- Language-Family Models Benefit Significantly: The language-family-focused BERTić model showed substantial performance gains when combined with the proposed data augmentation strategy, highlighting its effectiveness for specialized, low-resource models.
- Multilingual Models Excel: The instruction-tuned Aya-101 model, a 13B parameter multilingual model, demonstrated superior performance compared to smaller models, even with data augmentation.
- Script Consistency Matters: Discarding Cyrillic script data improved performance for certain languages and dialects when evaluating on Latin script test sets, indicating the importance of matching script usage between training and evaluation.
- Cross-Lingual Mix-and-Match is Inconclusive: While explored, the cross-lingual mix-and-match approach did not consistently lead to more language-agnostic models.
- Upsampling Doesn't Always Mean Better: Upsampling data from closely related languages or dialects did not always yield the best results for a specific target dialect, suggesting complex interactions in adaptation.
Key Results
- The GMUNLP submission achieved the highest scores across all three test datasets in the open-source model category.
- Their solution performed on par with the GPT-4 zero-shot iterative prompting approach, demonstrating competitiveness against state-of-the-art closed-source models.
- For the BERTić model, data augmentation led to substantial performance improvements.
- The Aya-101 model, particularly when fine-tuned with LoRA on augmented data, achieved the best average score across all languages in the open-source category. For instance, the Aya-101 model with LoRA tuning on the
otrslsetting achieved an average accuracy of 0.81. - The Torlak dialect was generally the easiest to predict, while the Cerkno dialect proved most challenging.
Limitations and Future Work
- Limited Synthetic Data Quality: While effective, the synthetic Chakavian data generated by Claude-3 showed limited accuracy in translation, indicating room for improvement in generation quality.
- Inconclusive Epoch Findings: Experiments with different numbers of training epochs (5 and 10) for instruction tuning Aya-101 yielded inconclusive findings.
- Future Directions: The authors suggest exploring advanced data augmentation techniques, investigating the impact of domain-specific knowledge integration, and developing novel architectures tailored to the unique characteristics of dialects. Further research into the complex interplay between language-relatedness and task-specific adaptation is also warranted.
Summary updated