TL;DR
The paper introduces KletterMix, a massive 725-billion-token German pretraining and annealing corpus constructed by translating a state-of-the-art English pretraining mixture (ClimbMix) into German. By preserving document boundaries, metadata, and topical diversity, KletterMix demonstrates that carefully translated data can transfer high-level curation structure and outperform models trained on native but noisier web crawls.
What Problem Does This Paper Address?
While English benefit from highly curated, diverse, and well-documented pretraining datasets, non-English resources—specifically German—remain underdeveloped. Existing German corpora are often noisy web crawls, small, or poorly validated subsets of multilingual datasets.
Simply crawling more native German text doesn't automatically replicate the sophisticated mixture design and topic coverage of top-tier English datasets. The authors explore an alternative: Can we successfully transfer the rich structural and topical diversity of a strong English pretraining corpus into German via high-quality machine translation, while avoiding common pitfalls like "translationese," semantic drift, and processing failures on long documents?
Methods
The authors developed a scalable, document-preserving translation pipeline to construct KletterMix from its English source, ClimbMix:
- Length-Aware Routing & Chunking: Documents are routed into context buckets based on length to optimize batching. Short documents are translated in a single pass, while long documents are split into sentences and greedily packed into chunks. To maintain discourse continuity, the prompt for each chunk includes a truncated window of the previous chunk's translation.
- Dynamic Target Budgeting: Instead of a fixed generation cap, the maximum target length scales dynamically with the source length to accommodate the natural expansion that occurs when translating English to German.
- Scalable Quality Estimation (QE): Because running expensive reference-free translation quality models across 725B tokens is computationally prohibitive, the authors used a two-tier approach. They scored a stratified pilot subset using COMETKiwi and used those scores to train a cheap, target-only gradient-boosted proxy model. This proxy model scores the full corpus using only target-side features (like GlotLID language confidence, token repetition, and character ratios), eliminating the need to reload the English source text during filtering.
Datasets
- ClimbMix: The high-quality English pretraining mixture used as the source corpus.
- KletterMix: The newly introduced German corpus containing 725B GPT-2 tokens, released under a CC-BY-NC-4.0 license.
- Baselines for Comparison: FineWeb2-DE and GermanWeb (established native German web corpora).
- Evaluation Suite: German and multilingual variants of MMLU (general knowledge), PIQA (physical reasoning), HellaSwag (commonsense inference), and ARC-Challenge (hard science QA).
Key Insights and Ideas
- Reasoning Transfer via Curation: KletterMix consistently outperforms native German web text on tasks requiring event continuation (HellaSwag) and compositional reasoning (ARC-C). This suggests that data curation and dense explanatory structure matter more for downstream reasoning than merely training on native surface forms.
- Target-Only Filtering Suffices: The success of the gradient-boosted proxy model proves that severe translation failures, wrong-language generation, and text degeneration can be caught cheaply at scale using only target-side text-shape and language identification features.
- Late-Stage Sharpening: When used during the "annealing" phase (late-stage training with a high-quality data mix), KletterMix acts as an excellent language-adaptation corpus. It boosts target-language performance significantly without degrading the model's original English capabilities.
Key Results
The authors evaluated KletterMix across two model scales:
- 0.6B Parameter Pretraining: A Qwen3-0.6B model trained from scratch on a 12B-token subset of KletterMix achieved lower training and validation loss than identical models trained on FineWeb2-DE or GermanWeb. The quality-filtered variant (KletterMix-Filt0.60) achieved the highest downstream performance with a 40.2% Core Average, beating FineWeb2-DE (38.3%) and GermanWeb (36.8%).
- 0.6B Corpus-Choice Annealing: Annealing a pretrained FineWeb2-DE model on KletterMix improved its performance by +1.1 percentage points over its baseline and +1.8 points over an identical branch annealed on GermanWeb.
- 7B Parameter Language-Mixture Annealing: The authors annealed an OLMo 3 7B model by replacing portions of English ClimbMix tokens with German KletterMix tokens (ranging from 0% to 20%). Increasing the KletterMix share led to a monotonic increase in German performance—boosting the four-task German mean by 4.5 percentage points at the 20% mixture level—while English performance remained completely stable (+0.3%).
Limitations and Future Work
Despite the strong results, KletterMix inherits the cultural, geographic, and topical biases of its original English sources. Machine translation can also introduce subtle semantic drift or unnatural stylistic choices ("translationese") that automatic proxy models might miss.
The authors suggest that future work should incorporate stricter filtering for boilerplate text and URLs, conduct deeper manual audits for factual preservation, and extend this document-preserving translation pipeline to other languages like French, Italian, and Spanish.
Summary updated