UAlign: LLM Alignment Benchmark for the Ukrainian Language

Andrian Kravchenko, Yurii Paniv, Nazarii DrushchakACL 2025

TL;DR

This paper introduces UAlign, the first comprehensive benchmark for evaluating the ethical alignment of Large Language Models (LLMs) in Ukrainian. By adapting existing English-language ethical datasets, UAlign reveals a performance gap between LLMs in Ukrainian and English, highlighting the need for more robust evaluation tools for low-resource languages.

What Problem Does This Paper Address?

The rapid advancement and widespread adoption of LLMs have raised significant concerns about their ethical behavior and safety. LLM alignment, the process of ensuring models are not only accurate but also safe, ethical, and aligned with human values, has become a critical research area. However, most existing alignment research and benchmarks are concentrated on high-resource languages like English and Chinese. This creates a risk for users of LLMs in other languages, as their specific cultural and ethical nuances may not be adequately addressed. The paper addresses this gap by developing a benchmark specifically for the Ukrainian language, enabling standardized evaluation of LLM ethical alignment in this context.

Methods

The authors developed the UAlign benchmark through a multi-phase process. First, they selected two established English-language datasets: ETHICS and Social Chemistry 101, chosen for their comprehensive sample collections and focus on classification tasks. These datasets underwent crowd-sourcing and rigorous human evaluation. For UAlign, they specifically selected the "commonsense" subset of ETHICS and isolated scenarios within the "care-harm" moral foundation from Social Chemistry 101.

The core of their methodology involved adapting these datasets for Ukrainian. This included:

  1. Machine Translation: Initially using the Dragoman model, they found its quality insufficient and subsequently adopted more advanced translation methods, ultimately selecting Claude 3.7 for its superior performance.
  2. Linguistic Refinement: After translation, the texts were refined using the Spivavtor model for grammatical error correction (GEC). This involved adjustments for first letter case, punctuation, and intrasentential modifications.
  3. Dataset Adaptation: The ETHICS "commonsense" subset was filtered to 1,700 shorter samples for efficient translation. The Social Chemistry 101 dataset was filtered based on inter-annotator agreement, focused on the "care-harm" foundation, deduplicated, and its five-point classification scale was simplified to a three-point scale.

Datasets

The UAlign benchmark is built upon two primary English-language datasets:

  • ETHICS: A dataset containing over 130,000 examples across five ethical domains. For UAlign, the "commonsense" subset was used, comprising 1,700 scenarios.
  • Social Chemistry 101: A corpus of 104,000 scenarios with 292,000 Rule-of-Thumb (RoT) judgments across five moral foundations. For UAlign, a filtered subset of 3,682 actions was used, focusing on the "care-harm" moral foundation and a simplified three-point judgment scale.

The paper also references other multilingual datasets like Aya Evaluation Suite and MultilingualHolisticBias, noting their limitations for Ukrainian evaluation due to translation quality or accessibility issues.

Key Insights and Ideas

  • Performance Gap: The evaluation of six LLMs revealed a notable performance gap between their alignment capabilities in Ukrainian compared to English. This underscores the challenges of cross-lingual alignment and the need for language-specific evaluation.
  • Translation Quality Matters: The authors emphasize that while machine translation has advanced, achieving sufficient quality for rigorous ethical evaluation requires careful selection of models and subsequent linguistic refinement. Their experience with Dragoman, DeepL, and Claude 3.7 highlights this.
  • Explicit Ukrainian Support Isn't Always Superior: Surprisingly, models that explicitly list Ukrainian support (like Aya models) did not consistently outperform general multilingual models on the UAlign benchmark, suggesting that explicit language support doesn't automatically guarantee superior ethical alignment.
  • Gemma 2's Strong Performance: Gemma 2, despite having a modest parameter count, showed results comparable to the proprietary GPT-40 across both benchmarks, indicating its strong multilingual capabilities.
  • Llama 3.2's Refusals: Llama 3.2 exhibited a significant number of refusals to answer prompts, particularly in Ukrainian, suggesting potential issues with its alignment or safety mechanisms in this language.

Key Results

The paper presents F1 scores for various LLMs on both the ETHICS and Social Chemistry 101 subsets of UAlign.

  • ETHICS Subset: GPT-40 achieved the highest F1 scores in both Ukrainian (0.631) and English (0.622), with Gemma 2 (Ukrainian: 0.772, English: 0.805) and Aya 101 (Ukrainian: 0.658, English: 0.612) also showing competitive results.
  • Social Chemistry 101 Subset: GPT-40 again performed strongly (Ukrainian: 0.631, English: 0.622), followed closely by Gemma 2 (Ukrainian: 0.668, English: 0.653). Notably, Llama 3.2 showed a significant drop in performance in Ukrainian (0.293) compared to English (0.608).

The paper also details the number of refusals for Llama 3.2, with 81 refusals on the ETHICS subset (Ukrainian) and 35 on the Social Chemistry 101 subset (Ukrainian), compared to fewer refusals in English.

Limitations and Future Work

Limitations: * Translation Quality: Despite advanced translation, the absence of comprehensive human verification means potential linguistic inaccuracies might still exist. * Cultural Scope: The source datasets primarily reflect English-speaking North American ethical scenarios, which may not universally apply to Ukrainian cultural contexts. * Representation Constraints: The adapted datasets cannot exhaustively cover the full spectrum of ethical scenarios relevant to Ukrainian society. * Methodological Simplification: The approach simplifies complex moral reasoning into discrete categories, potentially overlooking nuanced, contextual ethical judgments.

Future Work: * Expert Human Curation: Enhance benchmark quality through expert human curation and evaluation to improve translation accuracy and cultural relevance of ethical scenarios. * Expanded Scope: Extend the benchmark to encompass additional dimensions of value alignment beyond just ethical reasoning.

Summary updated

BibTeX
@inproceedings{kravchenko2025ualign,
  author = {Andrian Kravchenko and Yurii Paniv and Nazarii Drushchak},
  title = {{UAlign: LLM Alignment Benchmark for the Ukrainian Language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure