GottBERT: a pure German Language Model

Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, Martin BoekerEMNLP 2024 · DOI

Introduction

While Large Language Models (LLMs) like Llama 3 dominate current headlines, they are often computationally expensive and act as "jack-of-all-trades" regarding language. In many production and research environments, smaller, more efficient encoder-only models like BERT are preferred for specialized tasks such as Named Entity Recognition (NER) or text classification. GottBERT is the first dedicated German single-language RoBERTa model, pre-trained on a massive 145GB subset of the OSCAR corpus. It demonstrates that language-specific optimization and a custom vocabulary allow "smaller" BERT-style models to remain highly competitive against multilingual giants, often leading the pack in downstream German NLP tasks.

The authors identified a critical gap: while German BERT models existed, there was no dedicated German version of RoBERTa—a "robustly optimized" version of BERT that removes the Next Sentence Prediction task and utilizes larger batches and more data. Furthermore, the paper investigates a critical question in data engineering: does aggressive filtering of "noisy" web-crawled data actually improve model performance for German?

Methods and Architecture

The authors followed the RoBERTa training recipe but tailored it specifically for the German language. They trained both "Base" (125M parameters) and "Large" (355M parameters) versions using the fairseq library on Google TPU v3 and v4 pods.

Custom Tokenization

Instead of using an English-centric tokenizer, the team trained a custom Byte Pair Encoding (BPE) vocabulary of 52k tokens on 40GB of German text. This resulted in a 40% smaller binary data size compared to English-trained tokenizers, making the model more efficient at representing German text. This efficiency is crucial for handling German's complex compound words and unique characters.

Filtering Pipeline

A sophisticated cleaning process was developed to handle common web-crawl issues:

  • Umlaut Correction: Fixing decoding errors where characters like ä, ö, ü were corrupted.
  • Heuristic Filtering: Removing documents based on ratios of stopwords, punctuation, and capitalization.
  • SVM Filtering: Using a single-class Support Vector Machine (SVM) to identify and remove "meaningless" documents like word lists or source code.

Datasets and Comparison

The primary pre-training source was the German portion of the OSCAR dataset. The raw version was 145GB, while the filtered version was 121GB. GottBERT was benchmarked against several existing German and multilingual models to assess its relative performance.

Comparison of GottBERT with other German and multilingual models in terms of architecture, language count, and training data size.

Evaluation benchmarks included:

  • NLI: XNLI (German portion) for natural language inference.
  • NER: CoNLL 2003 and GermEval 2014.
  • Classification: GermEval 2018 (offensive language detection) and 10kGNAD (news article topic classification).

Key Results and Insights

GottBERT showed top-tier performance across the board, particularly in the "Base" category, where it often outperformed larger or more complex models.

Performance comparison of GottBERT variants against other models across several German NLP benchmarks.

Performance Highlights

  • NER Superiority: GottBERT Base led all other base models in both CoNLL 2003 and GermEval 2014 tasks.
  • Classification: On the 10kGNAD dataset, GottBERT models achieved ~90% accuracy, matching or exceeding much larger multilingual models like XLM-RoBERTa.
  • NLI: While GELECTRA slightly outperformed GottBERT in inference tasks, GottBERT remained a very strong second, achieving 82.46% accuracy in the Large version.

The Filtering Paradox

Surprisingly, the authors found that their rigorous filtering (removing ~24GB of data) did not significantly improve downstream performance. This suggests that modern transformer architectures are remarkably robust to "noisy" data, or that the "noise" actually provides useful variance that helps the model generalize better across diverse real-world text.

Vocabulary and Efficiency

By using a German-specific tokenizer, the model handles compound words and unique German characters much more effectively than multilingual models. Multilingual models often break German words into many tiny, less meaningful sub-tokens, whereas GottBERT's vocabulary is optimized for the language's structure. This highlights that for specific languages, a well-trained "small" model can punch far above its weight class.

Limitations and Future Work

Despite its success, GottBERT has limitations. Due to TPU library constraints at the time of training, the authors used 32-bit float precision instead of 16-bit, which increased memory usage and computation time. Additionally, like all BERT-style models, GottBERT is a "snapshot" of the data it was trained on and may struggle with very recent slang or cultural shifts without further fine-tuning.

Future work could benefit from incorporating more diverse sources beyond web crawls, such as legal documents (OpenLegalData) or Wikipedia, to improve the performance of the "Large" model variant.

Reference: Scheible, R., Frei, J., et al. (2024). GottBERT: a pure German Language Model. Proceedings of EMNLP 2024.

Summary updated

BibTeX
@inproceedings{scheible2024gottbert,
  author = {Raphael Scheible and Johann Frei and Fabian Thomczyk and Henry He and Patric Tippmann and Jochen Knaus and Victor Jaravine and Frank Kramer and Martin Boeker},
  title = {{GottBERT: a pure German Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1183},
}
Figure