Abstract
Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We opensource not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.
Introduction
In contemporary Natural Language Processing (NLP), Google’s multilingual BERT (mBERT) has long served as the primary resource for non-English languages. While mBERT covers over 100 languages, it is often limited by the "curse of multilinguality." By sharing a fixed vocabulary and parameter set across dozens of languages, these models frequently fail to capture the specific nuances, diacritics, and complex morphology of individual languages like Romanian.
This work introduces the first dedicated, high-performance transformer model for Romanian, built from the ground up to provide a robust baseline for the Romanian NLP community. By specializing in a single language, the model significantly outperforms multilingual alternatives across core tasks such as part-of-speech tagging, dependency parsing, and named entity recognition.
Training Corpus and Preprocessing
A critical component of a high-quality language model is the data it consumes. We compiled a massive 15.2 GB cleaned corpus from three primary sources: OPUS (translated web texts), OSCAR (filtered Common Crawl), and the Romanian Wikipedia dump.
Extensive cleaning was performed to handle Romanian-specific characters and hyphenation. This meticulous attention to data quality, particularly regarding diacritics, is a cornerstone of the model's performance and its ability to outperform models that treat Romanian as just one of many supported languages.
Model Architecture and Training
We followed the BERT-Base architecture, training both cased and uncased variants. Key technical details include:
- Vocabulary: A 50,000 word-piece vocabulary was generated using Byte-Pair Encoding (BPE). This language-specific vocabulary prevents common Romanian words from being fragmented into meaningless sub-units.
- Training Regime: The models were trained for 1 million steps on 8 Nvidia Volta V100 GPUs. We utilized a two-stage approach: 900,000 steps at a sequence length of 128, followed by 100,000 steps at 512 to ensure the model captures long-range dependencies.
- Optimization: We employed the LAMB optimizer, facilitating efficient convergence during large-batch training.
Tokenization Efficiency and Representation
One of the primary advantages of a monolingual model is tokenization efficiency. Romanian BERT encodes words in approximately 1.4 tokens on average, compared to mBERT's 2.0 tokens. This allows the model to process more context within its fixed 512-token window and develop deeper semantic representations.
To validate the quality of these representations, we conducted "frozen weight" tests, training only the final classification layer. Romanian BERT significantly outperformed mBERT in this setting, proving that its internal understanding of Romanian syntax and semantics is inherently superior to the multilingual alternative.
Experimental Results
Romanian BERT was evaluated against mBERT on several downstream tasks, including the Romanian Treebank (RRT) for POS tagging and parsing, and the RONEC dataset for Named Entity Recognition (NER).
- Part-of-Speech Tagging: The model achieved 98.18% accuracy on Universal POS (UPOS) tagging, showing a clear lead over mBERT.
- Dependency Parsing: We observed a Labeled Attachment Score (LAS) of 89.69%, surpassing mBERT’s 88.05%.
- Named Entity Recognition: The cased model reached an F1-score of 86.21% for entity typing, a 1.5% improvement over the multilingual baseline.
Conclusion and Future Work
The release of Romanian BERT provides a new state-of-the-art baseline for Romanian NLP. While the 15GB corpus is substantial, future iterations could benefit from even larger datasets and more advanced architectures like RoBERTa or ELECTRA. By open-sourcing the model and the training pipeline, we aim to foster further research and production-grade applications for the Romanian language.
Summary updated