JSI and WüNLP at the DIALECT-COPA Shared Task: In-Context Learning From Just a Few Dialectal Examples Gets You Quite Far

Nikola Ljubešić, Taja Kuzman, Peter Rupnik, Ivan Vulić, Fabian Schmidt, Goran Glavašš2024 · DOI

Abstract

The paper presents the JSI and WüNLP systems submitted to the DIALECT-COPA shared task on causal commonsense reasoning in dialectal texts. Jointly, we compare LLM-based zero-shot and few-shot in-context inference, and task-specific few-shot fine-tuning, in English and respective standard language, with zero-shot cross-lingual transfer (ZS-XLT) to the test dialects. Given the very strong zero-shot and especially few-shot in-context learning (ICL) performance, we further investigate whether task semantics, or language/dialect semantics explain the strong performance, showing that a significant part of the improvement indeed stems from learning the language or dialect semantics from the in-context examples, with only a minor contribution from understanding the nature of the task. The higher importance of the dialect semantics to the task semantics is further shown by the finding that the in-context learning with only a few dialectal instances achieves comparable results to the supervised fine-tuning approach on hundreds of instances in standard language.

Introduction

Causal commonsense reasoning—the ability to identify plausible causes and effects in everyday scenarios—is a hallmark of human-level intelligence. While Large Language Models (LLMs) demonstrate proficiency in this area for high-resource languages like English, their performance often falters when applied to "micro-dialects" such as Cerkno (Slovenian), Torlak (Serbian), or Chakavian (a surprise Croatian dialect). These varieties are underrepresented in the massive web-crawled datasets used for pre-training, making them a significant challenge for AI generalization.

This study utilizes the Choice Of Plausible Alternatives (COPA) dataset, translated into standard South Slavic languages (Slovenian, Croatian, Serbian, Macedonian) and their respective dialects. The goal was to determine if LLMs could bridge the gap between standard languages and these low-resource dialects using minimal data.

Methodology

The research team evaluated three primary strategies:

  1. Zero-Shot Inference: Models were tasked with solving dialectal reasoning problems using only an English task description.
  2. Few-Shot In-Context Learning (ICL): Models were provided with 10 dialectal examples within the prompt to guide their reasoning.
  3. Supervised Fine-Tuning (ZS-XLT): The Mixtral 8x7B model was fine-tuned using Q-LoRA on hundreds of examples in standard languages, then tested for "zero-shot" transfer to the dialects.

To understand the mechanics of ICL, the authors conducted Prompt Ablation Experiments. They compared a "List" prompt (providing dialectal sentences without task context) against a "Task" prompt (providing task structure without answers). This allowed them to isolate whether the model was learning the dialect's linguistic patterns or the logical structure of the COPA task.

Key Results

The results highlight a significant performance boost from few-shot learning across all models. GPT-4 emerged as the top performer, maintaining a strong lead over open-source alternatives.

Comparison of zero-shot vs. 10-shot performance across models.

As shown in the data, GPT-4's accuracy on dialects (DIA) improved from 76.0% in a zero-shot setting to 85.3% with just 10 examples. Similarly, Mixtral 8x7B saw its dialect performance rise from 52.1% to 62.4%. Interestingly, providing just 10 dialectal examples in a prompt yielded results comparable to fine-tuning a model on hundreds of instances in a standard language, demonstrating the extreme efficiency of ICL for low-resource scenarios.

Discussion and Insights

Dialect Semantics vs. Task Semantics

The ablation study revealed that the primary driver of ICL success is the model's exposure to the dialect's "semantics" (vocabulary and syntax) rather than the specific logic of the task.

Ablation study comparing task vs. dialect semantics across different varieties.

In Figure 1, the "list" prompt (focusing on dialectal examples) frequently outperformed or matched the "task" prompt. This suggests that once a model "recognizes" the linguistic variety, it can effectively apply its pre-existing reasoning capabilities. This "Power of 10" examples proves to be a robust bridge for low-resource linguistic transfer.

Proximity and Model Constraints

Linguistic proximity played a crucial role; models performed better on Torlak than on Cerkno, likely because Torlak shares more similarities with standard Serbian and Macedonian. Additionally, the study noted that smaller models like mT0 or Aya sometimes experienced performance degradation with more examples, pointing to limitations in context window capacity or encoder efficiency.

Conclusion

The findings demonstrate that LLMs can perform complex reasoning in extremely low-resource dialects with minimal intervention. While closed-source models like GPT-4 currently set the benchmark, open-source models like Mixtral show promise, especially when combined with cross-lingual fine-tuning. Future work will focus on refining these transfer strategies to support even more diverse and "unseen" linguistic varieties.

Summary updated

Abstract

The paper presents the JSI and WNLP systems submitted to the DIALECT-COPA shared task on causal commonsense reasoning in dialectal texts.Jointly, we compare LLM-based zeroshot and few-shot in-context inference (JSI team), and task-specific few-shot fine-tuning, in English and respective standard language, with zero-shot cross-lingual transfer (ZS-XLT) to the test dialects (WNLP team).Given the very strong zero-shot and especially few-shot in-context learning (ICL) performance, we further investigate whether task semantics, or language/dialect semantics explain the strong performance, showing that a significant part of the improvement indeed stems from learning the language or dialect semantics from the incontext examples, with only a minor contribution from understanding the nature of the task.The higher importance of the dialect semantics to the task semantics is further shown by the finding that the in-context learning with only a few dialectal instances achieves comparable results to the supervised fine-tuning approach on hundreds of instances in standard language.
BibTeX
@misc{ljubesic2024jsi,
  author = {Nikola Ljubešić and Taja Kuzman and Peter Rupnik and Ivan Vulić and Fabian Schmidt and Goran Glavašš},
  title = {{JSI and WüNLP at the DIALECT-COPA Shared Task: In-Context Learning From Just a Few Dialectal Examples Gets You Quite Far}},
  year = {2024},
  doi = {10.18653/v1/2024.vardial-1.18},
}
Figure