TL;DR
This paper introduces DIALECT-COPA, a new set of causal commonsense reasoning datasets for three South Slavic dialects: Cerkno Slovenian, Chakavian Croatian, and Torlak Serbian. These datasets extend the widely used COPA benchmark, enabling the evaluation of large language models (LLMs) on under-resourced dialects and revealing significant performance gaps compared to standard languages.
What Problem Does This Paper Address?
Evaluating the capabilities of large language models (LLMs) in understanding and reasoning about the world is crucial. The Choice of Plausible Alternatives (COPA) dataset has been a standard benchmark for causal commonsense reasoning. However, COPA was initially only available in English. While translations into several standard languages exist, there's a significant gap in evaluating LLMs on less-resourced languages and, importantly, on dialects. Dialects, often not well-represented in LLM training data, pose a unique challenge for generalization. This paper addresses this gap by creating dialectal versions of COPA for South Slavic languages, allowing for a more nuanced understanding of LLM performance beyond standard language varieties.
Methods
The authors extended the existing COPA dataset by translating its English original and corresponding standard language translations into three South Slavic dialects: Cerkno Slovenian, Chakavian Croatian, and Torlak Serbian. Native dialect speakers were employed as translators, with access to both the English original and the closest standard language translation to ensure authentic and accurate dialectal representations. This approach aimed to minimize translation artifacts and capture the nuances of each dialect. The paper also describes the challenges encountered during the translation process, such as non-canonical spelling and grammatical differences between dialects and standard languages.
Datasets
- DIALECT-COPA: This is the primary contribution of the paper, consisting of new COPA datasets for:
- Cerkno dialect of Slovenian
- Chakavian dialect of Croatian
- Torlak dialect of Serbian
- Standard South Slavic COPA Datasets: The paper also utilizes existing COPA datasets for standard Slovenian, Croatian, Serbian, and Macedonian for comparative analysis.
- English COPA Dataset: The original English COPA dataset served as the source for translations.
Key Insights and Ideas
- Dialects are a significant challenge for LLMs: The performance gap between standard languages and dialects is substantial, highlighting that LLMs struggle to generalize to non-standard language varieties.
- Dialectal similarity matters: The degree of similarity between a dialect and its corresponding standard language directly impacts LLM performance. The Torlak dialect, being closer to standard Serbian, showed better results than Cerkno Slovenian and Chakavian Croatian.
- In-context learning is powerful for dialects: Even with a limited number of examples (10-shot), in-context learning significantly boosts LLM performance on dialectal datasets, particularly for the most challenging ones.
- Translation challenges are real: Adapting datasets to dialects involves navigating issues like non-standard spelling and grammatical variations, requiring careful consideration and skilled translators.
Key Results
- Performance Gap: Baseline zero-shot experiments revealed a significant performance gap between standard languages and dialects. The Cerkno dialect was the most challenging, followed by Chakavian, while Torlak performed closest to its standard counterpart.
- Improvement with 10-shot learning: Few-shot learning (10-shot) dramatically improved LLM performance. The Cerkno dialect saw a 14-point improvement, and the Chakavian dialect improved by 9 points.
- Model Comparison: The closed GPT-4 model outperformed the open Mixtral model across all datasets and prompting scenarios.
- Quantitative Similarity: Character and word-level similarity analyses showed that the Torlak dialect is more similar to standard Serbian than the Cerkno and Chakavian dialects are to their respective standard languages.
Limitations and Future Work
- Written vs. Spoken Dialects: The authors acknowledge that the created datasets represent written projections of spoken dialects, which may not fully capture the nuances of spoken language.
- Canonicalization: Dialects lack a single canonical spelling, making translation and standardization challenging.
- Future Work: The authors plan to:
- Measure human performance on these dialectal datasets.
- Create speech audio datasets where native speakers read COPA instances, enabling evaluation of spoken dialectal language understanding and speech-to-text/speech-to-speech tasks.
- Encourage the creation of datasets for many more dialects.
Summary updated