KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report

Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, Michael Fromm, Nicolas Flores-Herr, Kristian KerstingarXiv 2026 · arXiv

TL;DR

The paper introduces KletterMix, a massive 725-billion-token German pretraining and annealing corpus constructed by translating a state-of-the-art English pretraining mixture (ClimbMix) into German. By preserving document boundaries, metadata, and topical diversity, KletterMix demonstrates that carefully translated data can transfer high-level curation structure and outperform models trained on native but noisier web crawls.

What Problem Does This Paper Address?

While English benefit from highly curated, diverse, and well-documented pretraining datasets, non-English resources—specifically German—remain underdeveloped. Existing German corpora are often noisy web crawls, small, or poorly validated subsets of multilingual datasets.

Simply crawling more native German text doesn't automatically replicate the sophisticated mixture design and topic coverage of top-tier English datasets. The authors explore an alternative: Can we successfully transfer the rich structural and topical diversity of a strong English pretraining corpus into German via high-quality machine translation, while avoiding common pitfalls like "translationese," semantic drift, and processing failures on long documents?

Methods

The authors developed a scalable, document-preserving translation pipeline to construct KletterMix from its English source, ClimbMix:

  1. Length-Aware Routing & Chunking: Documents are routed into context buckets based on length to optimize batching. Short documents are translated in a single pass, while long documents are split into sentences and greedily packed into chunks. To maintain discourse continuity, the prompt for each chunk includes a truncated window of the previous chunk's translation.
  2. Dynamic Target Budgeting: Instead of a fixed generation cap, the maximum target length scales dynamically with the source length to accommodate the natural expansion that occurs when translating English to German.
  3. Scalable Quality Estimation (QE): Because running expensive reference-free translation quality models across 725B tokens is computationally prohibitive, the authors used a two-tier approach. They scored a stratified pilot subset using COMETKiwi and used those scores to train a cheap, target-only gradient-boosted proxy model. This proxy model scores the full corpus using only target-side features (like GlotLID language confidence, token repetition, and character ratios), eliminating the need to reload the English source text during filtering.
Figure 1: Overview of the KletterMix pipeline. English source shards are routed into length-aware context buckets, translated with a document-preserving procedure that keeps document identifiers, order, and metadata, and executed shard-wise with parallel workers. A stratified sample of translations is scored with COMETKiwi to obtain reference-free quality labels, which supervise a cheap gradient- boosted proxy that scores the full corpus from target-side features and filters low-quality translations.

Datasets

  • ClimbMix: The high-quality English pretraining mixture used as the source corpus.
  • KletterMix: The newly introduced German corpus containing 725B GPT-2 tokens, released under a CC-BY-NC-4.0 license.
  • Baselines for Comparison: FineWeb2-DE and GermanWeb (established native German web corpora).
  • Evaluation Suite: German and multilingual variants of MMLU (general knowledge), PIQA (physical reasoning), HellaSwag (commonsense inference), and ARC-Challenge (hard science QA).

Key Insights and Ideas

  • Reasoning Transfer via Curation: KletterMix consistently outperforms native German web text on tasks requiring event continuation (HellaSwag) and compositional reasoning (ARC-C). This suggests that data curation and dense explanatory structure matter more for downstream reasoning than merely training on native surface forms.
  • Target-Only Filtering Suffices: The success of the gradient-boosted proxy model proves that severe translation failures, wrong-language generation, and text degeneration can be caught cheaply at scale using only target-side text-shape and language identification features.
  • Late-Stage Sharpening: When used during the "annealing" phase (late-stage training with a high-quality data mix), KletterMix acts as an excellent language-adaptation corpus. It boosts target-language performance significantly without degrading the model's original English capabilities.

Key Results

The authors evaluated KletterMix across two model scales:

  • 0.6B Parameter Pretraining: A Qwen3-0.6B model trained from scratch on a 12B-token subset of KletterMix achieved lower training and validation loss than identical models trained on FineWeb2-DE or GermanWeb. The quality-filtered variant (KletterMix-Filt0.60) achieved the highest downstream performance with a 40.2% Core Average, beating FineWeb2-DE (38.3%) and GermanWeb (36.8%).
  • 0.6B Corpus-Choice Annealing: Annealing a pretrained FineWeb2-DE model on KletterMix improved its performance by +1.1 percentage points over its baseline and +1.8 points over an identical branch annealed on GermanWeb.
  • 7B Parameter Language-Mixture Annealing: The authors annealed an OLMo 3 7B model by replacing portions of English ClimbMix tokens with German KletterMix tokens (ranging from 0% to 20%). Increasing the KletterMix share led to a monotonic increase in German performance—boosting the four-task German mean by 4.5 percentage points at the 20% mixture level—while English performance remained completely stable (+0.3%).
Figure 2: Proxy-score distribution and filtering thresholds used to construct the three 12B-token filtering ablations. The histogram shows the full translated-corpus proxy-score distribution on a log-count axis. Dashed vertical lines mark the retained-document thresholds at 0.50, 0.55, and 0.60; colored regions indicate which documents are removed or retained by progressively stricter filters.

Limitations and Future Work

Despite the strong results, KletterMix inherits the cultural, geographic, and topical biases of its original English sources. Machine translation can also introduce subtle semantic drift or unnatural stylistic choices ("translationese") that automatic proxy models might miss.

The authors suggest that future work should incorporate stricter filtering for boilerplate text and URLs, conduct deeper manual audits for factual preservation, and extend this document-preserving translation pipeline to other languages like French, Italian, and Spanish.

Summary updated

Abstract

High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their English counterparts: they are often smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. We introduce KletterMix, a high-quality German corpus for language model pretraining and annealing, designed as a reusable dataset artifact for the natural language processing and modeling community. KletterMix is built by translating a state-of-the-art English pretraining corpus into German while preserving document boundaries, metadata, source structure, and topical diversity. This construction yields a German corpus with the scale and diversity of a modern pretraining dataset, while enabling direct comparison to its English source. We document the dataset through a broad set of corpus-level analyses, including translation quality, document length distributions, topic coverage, source composition, and geographic metadata. Using COMETKiwi, we show that the translated documents achieve strong quality across diverse domains, suggesting that careful translation can preserve much of the semantic and stylistic richness of the original corpus. Beyond dataset construction, we evaluate KletterMix as training data. Through controlled pretraining and annealing ablations against established German corpora, we show that models trained on KletterMix achieve measurable improvements on German-language downstream evaluations. These results demonstrate that carefully curated translated data can substantially strengthen the German pretraining data ecosystem.
BibTeX
@misc{kraus2026klettermix,
  author = {Maurice Kraus and Ruben Härle and Sebastian Sztwiertnia and Abbas Goher Khan and Mehdi Ali and Michael Fromm and Nicolas Flores-Herr and Kristian Kersting},
  title = {{KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report}},
  year = {2026},
  eprint = {2606.03773},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.03773},
}
Figure