The UNLP 2024 Shared Task on Fine-Tuning Large Language Models for Ukrainian

Mariana Romanyshyn, Oleksiy Syvokon, Roman KyslyiCOLING 2024

TL;DR

This paper introduces the first shared task focused on fine-tuning large language models (LLMs) for Ukrainian. It addresses the critical need for LLMs that understand Ukrainian language, history, and culture, and presents the results of this novel competition, highlighting the challenges and successes in adapting LLMs for a less-resourced language.

What Problem Does This Paper Address?

Large Language Models (LLMs) have revolutionized natural language processing, but their development has largely centered on high-resource languages like English. This leaves many languages, including Ukrainian, with limited access to sophisticated LLM capabilities. Training LLMs from scratch is computationally prohibitive for most researchers, and adapting existing models requires specialized datasets and methodologies. The UNLP 2024 Shared Task aimed to bridge this gap by facilitating the creation of LLMs specifically fine-tuned for Ukrainian, focusing on knowledge of its language, history, and culture, while ensuring reproducibility and reasonable resource usage.

Methods

The shared task employed a two-pronged evaluation approach:

  • Automated Evaluation: A set of multiple-choice exam questions, derived from the Ukrainian External Independent Evaluation (ZNO), was used to assess LLMs' factual knowledge. Accuracy was the primary metric.
  • Human Evaluation: For open-ended text generation tasks, a more subjective evaluation was conducted. Participants' model outputs were presented side-by-side anonymously, and human evaluators (native Ukrainian speakers) voted on which response was better. The TrueSkill ranking algorithm was used to aggregate these judgments and rank the models.

Participants were restricted to using open-weight LLMs (e.g., Llama 2, Mistral 7B, Phi-2, Gemma, Aya 101) that could run on GPUs with 16GB VRAM. They were allowed to use various fine-tuning techniques, prompting strategies (like few-shot learning or chain-of-thought), and retrieval-augmented generation (RAG). The emphasis was on reproducible solutions and making the final models publicly available.

Datasets

The shared task utilized two primary datasets:

  • Exam Questions: This dataset comprised machine-readable multiple-choice questions and answers from the Ukrainian External Independent Evaluation (ZNO) for History of Ukraine and Ukrainian language and literature, covering years 2006-2023. The training set had 3,063 questions, and the test set had 751.
  • Open Questions: This dataset was curated by native Ukrainian speakers and included a mix of common knowledge questions, composition tasks, rewrite tasks, and evaluation tasks. It was formatted in the Alpaca dataset style. The dev set contained 20 questions, and the hidden test set had 100.

Key Insights and Ideas

  • The "Open LLM" Advantage: The task successfully demonstrated that open-weight LLMs, when appropriately fine-tuned, can achieve competitive performance for specific languages, even when compared to proprietary models.
  • Data Scarcity is a Major Hurdle: A recurring theme among participants was the scarcity of readily available, high-quality open datasets for Ukrainian. This forced teams to be creative in data augmentation and synthesis.
  • Reproducibility is Key: The task's strict limitations on model size and accessibility were crucial for ensuring that the results were reproducible and could serve as a foundation for future research.
  • Human Evaluation Remains Vital: For nuanced tasks like text generation, human evaluation, despite its cost, remains indispensable for capturing subjective quality and user experience.

Key Results

Three teams submitted solutions: Sherlock, CodeKobzar, and UkraineNow.

  • Exam Questions Task: The top-performing system, Sherlock (with RAG), achieved an accuracy of 49%. The other Sherlock submission (no RAG) scored 42%, CodeKobzar 39%, and UkraineNow 23%.
  • Open Questions Task: Using the TrueSkill ranking system, Sherlock (no RAG) achieved the highest rating (26.77), followed by Sherlock (RAG) (26.27), UkraineNow (24.89), and CodeKobzar (23.79).
  • Comparison with GPT-4: For the exam task, GPT-4 achieved an accuracy of 0.61 with a simple prompt. For the open questions task, GPT-4's performance, while not explicitly quantified in a single score, was generally superior to the winning open-source solutions, indicating room for improvement in the fine-tuned models.

Limitations and Future Work

The paper acknowledges several limitations and suggests future directions:

  • Limited Number of Submissions: Only three teams submitted solutions, which might not fully represent the breadth of research in this area.
  • Open Questions Evaluation: While human evaluation was used, the paper suggests introducing automated metrics for open questions in future iterations to complement human judgment.
  • Dataset Size and Variability: The authors plan to increase the size and variability of test sets for future shared tasks.
  • Comparison with Proprietary Models: The task was limited to open models to ensure reproducibility. Future work could explore more direct comparisons with proprietary models on a wider range of tasks.
  • Code-Switching: One team (UkraineNow) highlighted the phenomenon of code-switching in Ukrainian, suggesting it's an important area for future LLM adaptation.

Summary updated

BibTeX
@inproceedings{romanyshyn2024unlp,
  author = {Mariana Romanyshyn and Oleksiy Syvokon and Roman Kyslyi},
  title = {{The UNLP 2024 Shared Task on Fine-Tuning Large Language Models for Ukrainian}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure