TL;DR
Researchers from GMUNLP have developed a robust framework for adapting Large Language Models (LLMs) to South Slavic micro-dialects using advanced data augmentation. By combining synthetic data generation from closed-source models with specialized fine-tuning of open-source models, they achieved the top-ranking performance in the open-source category for the DIALECT-COPA reasoning task, even rivaling zero-shot GPT-4.
What Problem Does This Paper Address?
While LLMs have become remarkably proficient at reasoning in standard languages (like English or standard Croatian), they often stumble when faced with non-standard dialectal varieties. This "dialect gap" is primarily a data scarcity problem: there simply isn't enough high-quality training data for micro-dialects to allow models to capture their unique linguistic nuances.
The authors participate in the DIALECT-COPA shared task, which evaluates commonsense reasoning—specifically the "Choice of Plausible Alternatives" (COPA). In this task, a model must determine the most likely cause or effect of a given premise. The challenge here is doing so in three specific South Slavic micro-dialects: Chakavian, Cerkno, and Torlak.
Methods
The authors propose a multi-phase approach that leverages the strengths of different model architectures and data generation strategies:
- Model Selection: They compared several base models, ultimately selecting BERTić (a smaller, language-family-focused encoder model) and Aya-101 (a mid-sized, 13B parameter multilingual instruction-tuned model) based on preliminary validation performance.
- Data Augmentation: To combat the lack of dialectal data, they employed several clever tricks:
- Transliteration: Converting Macedonian and Serbian Cyrillic data to Latin script to maintain consistency across the dataset.
- Reverse-Augmentation: Doubling the training set by swapping the premise and the correct answer choice (e.g., turning a "cause" question into an "effect" question).
- Synthetic Generation: Using Claude-3 to translate standard Croatian into the Chakavian dialect by providing it with specific grammar rules and few-shot examples. They also used GPT-4 to generate synthetic English COPA data, which was then machine-translated into other target languages using the NLLB-6B model.
- Task-Specific Tuning: They used LoRA (Low-Rank Adaptation) for the larger Aya-101 model to ensure efficient tuning on small datasets, while performing full fine-tuning on the smaller BERTić model.
Datasets
The study utilizes the DIALECT-COPA dataset, which includes training and validation data for seven languages/dialects (English, Slovenian, Cerkno, Croatian, Serbian, Torlak, and Macedonian) and a test set featuring two micro-dialects (Cerkno, Torlak) plus one previously unseen dialect (Chakavian).
Key Insights and Ideas
- The Power of Synthetic Data: Even though the Claude-3 translations for Chakavian were only partially accurate (correctly translating only a few key words per sentence), they provided enough dialectal "signal" to significantly boost model performance.
- Script Consistency Matters: The authors found that using only Latin script data for training improved performance on Latin-script evaluations, suggesting that script variations can be a distractor for models in low-resource settings.
- Cross-Lingual Synergy: Combining datasets from multiple related languages consistently outperformed training on a single language, confirming that LLMs can leverage shared linguistic structures across the South Slavic family.
- Instruction Tuning vs. Scale: The instruction-tuned Aya-101 (13B) consistently outperformed the smaller, language-specific BERTić, highlighting the value of general instruction-following capabilities even in niche dialectal tasks.
Key Results
The GMUNLP team achieved the highest scores across all three test datasets in the open-source model category. Notably, their Aya-101 model tuned with LoRA performed on par with a zero-shot iterative prompting approach using GPT-4.
As shown in the preliminary results (Figure 2), BERTić showed a strong affinity for Croatian (64% accuracy), but the data augmentation strategy was the "secret sauce" that allowed these models to generalize to the micro-dialects in the final test phase.
Limitations and Future Work
The authors noted that while synthetic data was helpful, the translation quality from models like Claude-3 for specific dialects remains limited. Furthermore, their experiments with different training durations (5 vs. 10 epochs) yielded inconclusive results, suggesting that the optimal "sweet spot" for tuning on such small, augmented datasets is still unclear.
Future directions include exploring more advanced data augmentation techniques, integrating domain-specific knowledge, and developing novel architectures specifically designed to handle the unique characteristics of dialects.
Summary updated