Introduction
While benchmarks like GLUE and SuperGLUE have catalyzed advancements in English Natural Language Processing (NLP), many other languages continue to lack unified evaluation frameworks. Bulgarian, an Eastern South-Slavic language, has historically faced challenges due to fragmented datasets, non-standardized evaluation splits, and potential data leakage. To address these issues, we introduce bgGLUE, the first comprehensive benchmark for Bulgarian Natural Language Understanding (NLU). bgGLUE provides a centralized, standardized suite of nine diverse tasks to facilitate the systematic evaluation and comparison of language models.
Methodology
The development of bgGLUE focused on curating a representative set of NLU challenges while ensuring rigorous evaluation standards. A primary contribution is the standardization of data splits. Rather than relying on random splits, which can lead to models memorizing specific entities or temporal events, the researchers implemented topic-based or temporal splits. This approach ensures that models are evaluated on their ability to learn general linguistic patterns rather than specific dataset artifacts.
To establish strong baselines, the following models were evaluated:
- Multilingual Pre-trained Language Models (PLMs): mBERT and XLM-R (Base and Large).
- Knowledge-Distilled Models: Distil-mBERT and MiniLM, evaluated to understand the trade-offs between efficiency and performance.
- Domain-Specific Models: SlavicBERT, pre-trained specifically on Slavic languages.
Benchmark Composition
The bgGLUE benchmark consists of nine datasets spanning various domains, including news, Wikipedia, social media, and academic examinations. These tasks are categorized into token classification, regression/ranking, and general text classification.
Key tasks include sequence labeling (NER and POS tagging), sentiment analysis (Cinexio), check-worthiness ranking (CT21.T1), and complex reasoning tasks such as Natural Language Inference (XNLI) and multiple-choice question answering (EXAMS).
Evaluation and Results
The evaluation provides a clear hierarchy of current model capabilities for Bulgarian. Models were fine-tuned on each task individually, and a composite "bgGLUE score" was calculated to provide an aggregate performance metric.
Key Findings
- Top Performance and Scaling: XLM-R Large emerged as the top performer with an overall bgGLUE score of 75.82. The results demonstrate a clear correlation between model size and performance, with larger models consistently leading across almost all tasks.
- The "Reasoning Gap": While models achieved near-perfect scores on sequence labeling (e.g., ~99% on POS tagging), they struggled significantly with reasoning-heavy tasks. The EXAMS dataset proved the most difficult, with the best model achieving only 36.41% accuracy, highlighting a significant area for future improvement.
- Efficiency of Distillation: Distilled models like MiniLM proved highly competitive, often performing within 2-3 points of their larger teacher models. This suggests that distilled models are viable for resource-constrained Bulgarian NLP applications.
- Domain-Specific vs. Multilingual: SlavicBERT showed strength in NER, likely due to its specific pre-training, but did not consistently outperform general multilingual models on broader reasoning tasks.
Conclusion and Future Work
bgGLUE establishes a necessary foundation for Bulgarian NLU, revealing that while current models excel at surface-level linguistic tasks, a substantial gap remains in complex reasoning. This study focused on encoder-only models under one billion parameters. Future research will expand the benchmark to include generative (decoder-only) models, larger-scale architectures, and multi-modal tasks. Additionally, investigating model biases and safety in Bulgarian-specific contexts remains a critical priority for the development of robust Bulgarian NLP systems.
Summary updated