Do LLMs Speak Kazakh? A Pilot Evaluation of Seven Models

Akylbek Maxutov, Ayan Myrzakhmet, Pavel BraslavskiACL 2024

TL;DR

This paper presents a systematic evaluation of seven Large Language Models (LLMs) on various tasks in Kazakh, a Turkic language. The study reveals that while LLMs show promise, their performance on Kazakh lags behind English, with GPT-4 leading the pack. The findings offer crucial insights into the current capabilities and limitations of LLMs for low-resource languages.

What Problem Does This Paper Address?

Large Language Models (LLMs) are rapidly transforming various industries, but their development has largely focused on high-resource languages like English. This creates a potential "inequality gap" for speakers of languages with fewer digital resources. Training LLMs for these "low-resource" languages is challenging due to data scarcity and economic hurdles. While some multilingual LLMs exhibit capabilities in other languages, their performance is often suboptimal. This paper addresses the critical need to evaluate LLMs in languages like Kazakh, which, despite having millions of speakers, is underrepresented in LLM research. The authors aim to provide a foundational understanding of how current LLMs perform on Kazakh tasks and to inform future development for similar languages.

Methods

The researchers conducted a systematic, automatic benchmark-based evaluation of seven LLMs. They employed a zero-shot scenario, meaning the models were given instructions and tasks without prior specific training on those tasks. Two types of prompts were used: English and Kazakh instructions. For classification tasks, simple scripts were used to extract answers. For generative tasks like open-book QA and machine translation, they used standard metrics like F1 and BLEU scores. Spelling correction was evaluated using token-level Jaccard coefficients. For closed-book QA, they used token-level recall to account for the flexibility of LLM responses.

Datasets

The study utilized a diverse set of six datasets, with three being newly prepared for this research:

  • Belebele: A multilingual multiple-choice QA dataset, with Kazakh translations.
  • kkCOPA: A machine-translated version of the English Choice Of Plausible Alternatives (COPA) dataset, focusing on causal reasoning.
  • NIS Math: Original math problems in Kazakh, scraped and post-processed from entrance tests.
  • KazQAD: An open-domain question answering dataset in Kazakh, used in both open- and closed-book scenarios.
  • kkWikiSpell: A newly created Kazakh spelling correction dataset, derived from Wikipedia pages.
  • FLORES-101: A dataset for machine translation evaluation, used here to evaluate Kazakh to English, Russian, and Turkish translation.

Key Insights and Ideas

  • Performance Gap: As expected, LLMs perform significantly worse on Kazakh tasks compared to their English counterparts, highlighting the challenges of low-resource language adaptation.
  • Classification vs. Generation: LLMs generally excel at classification tasks but struggle with generative tasks, a trend observed across both English and Kazakh.
  • Prompting Matters (but not always as expected): While English prompts sometimes improve performance, the study found mixed results, and in some cases, Kazakh prompts yielded better scores. The hypothesis that English prompts systematically improve LLM quality on non-English tasks was not consistently confirmed.
  • Model Size vs. Performance: The open-source model AYA, despite its smaller size, showed competitive performance, suggesting that architectural advancements and training data quality can be as crucial as sheer parameter count.
  • Tokenization Inequity: The paper touches upon how tokenization can impact LLM costs and potentially create inequities between language communities, as APIs often charge per token.

Key Results

  • GPT-4 Dominance: GPT-4 emerged as the top-performing model across most tasks, demonstrating superior capabilities in Kazakh.
  • Gemini and AYA as Strong Contenders: Gemini and AYA followed GPT-4, showing promising results, especially considering AYA's smaller size.
  • Translation Quality: Dedicated machine translation solutions still outperform general LLMs for Kazakh, though GPT-4 is approaching Google Translate's quality on the Kazakh-Turkish pair.
  • Spelling Correction Challenges: The spelling correction task proved difficult for all models, with GPT-4 leading.
  • Open-Book QA: AYA surprisingly outperformed GPT-4 and Gemini on open-book QA with English prompts.
  • LLAMA 2 Performance: LLAMA 2's performance was generally lower than expected, potentially due to its quantized 7B version and less optimal prompting strategies.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Resources: Due to budget constraints, they could not create extensive datasets or manually translate many existing ones. The KazQAD test set was also sampled to manage costs.
  • Data Contamination: There's a possibility that some models may have encountered the KazQAD test set during pre-training, as it was recently released.
  • Model Specifics: For closed-source models, details like parameter count and training data remain unknown, making direct comparisons challenging.
  • Prompt Engineering: While they used both English and Kazakh prompts, further extensive prompt engineering could potentially yield different results.

Future work suggested includes expanding the list of evaluated LLMs and increasing the diversity and size of the benchmark datasets. The authors also plan to investigate the discrepancy in LLAMA 2's performance and explore more advanced prompt engineering techniques. The released datasets and code are intended to serve as a foundation for future LLM research in Kazakh.

Summary updated

BibTeX
@inproceedings{maxutov2024do,
  author = {Akylbek Maxutov and Ayan Myrzakhmet and Pavel Braslavski},
  title = {{Do LLMs Speak Kazakh? A Pilot Evaluation of Seven Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure