bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark

Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Ves Stoyanov, Ivan Koychev, Preslav Nakov, Dragomir Radev

Introduction

While benchmarks like GLUE and SuperGLUE have catalyzed advancements in English Natural Language Processing (NLP), many other languages continue to lack unified evaluation frameworks. Bulgarian, an Eastern South-Slavic language, has historically faced challenges due to fragmented datasets, non-standardized evaluation splits, and potential data leakage. To address these issues, we introduce bgGLUE, the first comprehensive benchmark for Bulgarian Natural Language Understanding (NLU). bgGLUE provides a centralized, standardized suite of nine diverse tasks to facilitate the systematic evaluation and comparison of language models.

Methodology

The development of bgGLUE focused on curating a representative set of NLU challenges while ensuring rigorous evaluation standards. A primary contribution is the standardization of data splits. Rather than relying on random splits, which can lead to models memorizing specific entities or temporal events, the researchers implemented topic-based or temporal splits. This approach ensures that models are evaluated on their ability to learn general linguistic patterns rather than specific dataset artifacts.

To establish strong baselines, the following models were evaluated:

  • Multilingual Pre-trained Language Models (PLMs): mBERT and XLM-R (Base and Large).
  • Knowledge-Distilled Models: Distil-mBERT and MiniLM, evaluated to understand the trade-offs between efficiency and performance.
  • Domain-Specific Models: SlavicBERT, pre-trained specifically on Slavic languages.

Benchmark Composition

The bgGLUE benchmark consists of nine datasets spanning various domains, including news, Wikipedia, social media, and academic examinations. These tasks are categorized into token classification, regression/ranking, and general text classification.

Summary of the nine tasks included in the bgGLUE benchmark, categorized by type, metrics, and domain.

Key tasks include sequence labeling (NER and POS tagging), sentiment analysis (Cinexio), check-worthiness ranking (CT21.T1), and complex reasoning tasks such as Natural Language Inference (XNLI) and multiple-choice question answering (EXAMS).

Evaluation and Results

The evaluation provides a clear hierarchy of current model capabilities for Bulgarian. Models were fine-tuned on each task individually, and a composite "bgGLUE score" was calculated to provide an aggregate performance metric.

Baseline results on the bgGLUE benchmark across all nine tasks, including random baselines and fine-tuning results for various PLMs.

Key Findings

  • Top Performance and Scaling: XLM-R Large emerged as the top performer with an overall bgGLUE score of 75.82. The results demonstrate a clear correlation between model size and performance, with larger models consistently leading across almost all tasks.
  • The "Reasoning Gap": While models achieved near-perfect scores on sequence labeling (e.g., ~99% on POS tagging), they struggled significantly with reasoning-heavy tasks. The EXAMS dataset proved the most difficult, with the best model achieving only 36.41% accuracy, highlighting a significant area for future improvement.
  • Efficiency of Distillation: Distilled models like MiniLM proved highly competitive, often performing within 2-3 points of their larger teacher models. This suggests that distilled models are viable for resource-constrained Bulgarian NLP applications.
  • Domain-Specific vs. Multilingual: SlavicBERT showed strength in NER, likely due to its specific pre-training, but did not consistently outperform general multilingual models on broader reasoning tasks.

Conclusion and Future Work

bgGLUE establishes a necessary foundation for Bulgarian NLU, revealing that while current models excel at surface-level linguistic tasks, a substantial gap remains in complex reasoning. This study focused on encoder-only models under one billion parameters. Future research will expand the benchmark to include generative (decoder-only) models, larger-scale architectures, and multi-modal tasks. Additionally, investigating model biases and safety in Bulgarian-specific contexts remains a critical priority for the development of robust Bulgarian NLP systems.

Summary updated

BibTeX
@inproceedings{hardalov2023bgglue,
  author = {Momchil Hardalov and Pepa Atanasova and Todor Mihaylov and Galia Angelova and Kiril Simov and Petya Osenova and Ves Stoyanov and Ivan Koychev and Preslav Nakov and Dragomir Radev},
  title = {{bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.487},
}
Figure