Data-Augmentation-Based Dialectal Adaptation for LLMs

Fahim Faisal, Antonios Anastasopoulos

TL;DR

Researchers from GMUNLP have developed a robust framework for adapting Large Language Models (LLMs) to South Slavic micro-dialects using advanced data augmentation. By combining synthetic data generation from closed-source models with specialized fine-tuning of open-source models, they achieved the top-ranking performance in the open-source category for the DIALECT-COPA reasoning task, even rivaling zero-shot GPT-4.

What Problem Does This Paper Address?

While LLMs have become remarkably proficient at reasoning in standard languages (like English or standard Croatian), they often stumble when faced with non-standard dialectal varieties. This "dialect gap" is primarily a data scarcity problem: there simply isn't enough high-quality training data for micro-dialects to allow models to capture their unique linguistic nuances.

The authors participate in the DIALECT-COPA shared task, which evaluates commonsense reasoning—specifically the "Choice of Plausible Alternatives" (COPA). In this task, a model must determine the most likely cause or effect of a given premise. The challenge here is doing so in three specific South Slavic micro-dialects: Chakavian, Cerkno, and Torlak.

Methods

The authors propose a multi-phase approach that leverages the strengths of different model architectures and data generation strategies:

  1. Model Selection: They compared several base models, ultimately selecting BERTić (a smaller, language-family-focused encoder model) and Aya-101 (a mid-sized, 13B parameter multilingual instruction-tuned model) based on preliminary validation performance.
  2. Data Augmentation: To combat the lack of dialectal data, they employed several clever tricks:
  3. Transliteration: Converting Macedonian and Serbian Cyrillic data to Latin script to maintain consistency across the dataset.
  4. Reverse-Augmentation: Doubling the training set by swapping the premise and the correct answer choice (e.g., turning a "cause" question into an "effect" question).
  5. Synthetic Generation: Using Claude-3 to translate standard Croatian into the Chakavian dialect by providing it with specific grammar rules and few-shot examples. They also used GPT-4 to generate synthetic English COPA data, which was then machine-translated into other target languages using the NLLB-6B model.
  6. Task-Specific Tuning: They used LoRA (Low-Rank Adaptation) for the larger Aya-101 model to ensure efficient tuning on small datasets, while performing full fine-tuning on the smaller BERTić model.
Training data augmentation approaches

Datasets

The study utilizes the DIALECT-COPA dataset, which includes training and validation data for seven languages/dialects (English, Slovenian, Cerkno, Croatian, Serbian, Torlak, and Macedonian) and a test set featuring two micro-dialects (Cerkno, Torlak) plus one previously unseen dialect (Chakavian).

DIALECT-COPA dataset statistics

Key Insights and Ideas

  • The Power of Synthetic Data: Even though the Claude-3 translations for Chakavian were only partially accurate (correctly translating only a few key words per sentence), they provided enough dialectal "signal" to significantly boost model performance.
  • Script Consistency Matters: The authors found that using only Latin script data for training improved performance on Latin-script evaluations, suggesting that script variations can be a distractor for models in low-resource settings.
  • Cross-Lingual Synergy: Combining datasets from multiple related languages consistently outperformed training on a single language, confirming that LLMs can leverage shared linguistic structures across the South Slavic family.
  • Instruction Tuning vs. Scale: The instruction-tuned Aya-101 (13B) consistently outperformed the smaller, language-specific BERTić, highlighting the value of general instruction-following capabilities even in niche dialectal tasks.

Key Results

The GMUNLP team achieved the highest scores across all three test datasets in the open-source model category. Notably, their Aya-101 model tuned with LoRA performed on par with a zero-shot iterative prompting approach using GPT-4.

Performance comparison on DIALECT-COPA test set

As shown in the preliminary results (Figure 2), BERTić showed a strong affinity for Croatian (64% accuracy), but the data augmentation strategy was the "secret sauce" that allowed these models to generalize to the micro-dialects in the final test phase.

Limitations and Future Work

The authors noted that while synthetic data was helpful, the translation quality from models like Claude-3 for specific dialects remains limited. Furthermore, their experiments with different training durations (5 vs. 10 epochs) yielded inconclusive results, suggesting that the optimal "sweet spot" for tuning on such small, augmented datasets is still unclear.

Future directions include exploring more advanced data augmentation techniques, integrating domain-specific knowledge, and developing novel architectures specifically designed to handle the unique characteristics of dialects.

Summary updated

Abstract

This report presents GMUNLP's participation to the Dialect-Copa shared task at VarDial 2024, which focuses on evaluating the commonsense reasoning capabilities of large language models (LLMs) on South Slavic micro-dialects. The task aims to assess how well LLMs can handle non-standard dialectal varieties, as their performance on standard languages is already well-established. We propose an approach that combines the strengths of different types of language models and leverages data augmentation techniques to improve task performance on three South Slavic dialects: Chakavian, Cherkano, and Torlak. We conduct experiments using a language-family-focused encoder-based model (BERTić) and a domain-agnostic multilingual model (AYA-101). Our results demonstrate that the proposed data augmentation techniques lead to substantial performance gains across all three test datasets in the open-source model category. This work highlights the practical utility of data augmentation and the potential of LLMs in handling non-standard dialectal varieties, contributing to the broader goal of advancing natural language understanding in low-resource and dialectal settings. Code:https://github.com/ffaisal93/dialect_copa
BibTeX
@inproceedings{faisal2024data,
  author = {Fahim Faisal and Antonios Anastasopoulos},
  title = {{Data-Augmentation-Based Dialectal Adaptation for LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure