Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Felicia Körner, Maria Matveev, Florian Eichin, Gitta Kutyniok, Barbara Plank, Michael A. HeddericharXiv 2026 · arXiv

TL;DR

How do large language models (LLMs) learn to translate without explicit translation training? By saving high-density checkpoints of a 1.7B parameter model from scratch, researchers discovered that cross-lingual capabilities emerge in two distinct phases: the model first relies on an overeager token-copying strategy before gradually developing a more abstract, generalizing translation mechanism.

What Problem Does This Paper Address?

Multilingual LLMs possess a remarkable ability called cross-lingual transfer—they can generalize knowledge across languages even without explicit parallel training data or cross-lingual objectives. While researchers know this happens, how it emerges during the early stages of pretraining has remained a black box.

Prior studies typically analyze fully trained models or look at very sparse checkpoints, missing the fine-grained behavioral and internal shifts that happen early on. This paper addresses this gap by tracking the step-by-step trajectory of a model's internal mechanics during its first few billion tokens of training to understand how unsupervised translation actually develops.

Methods

The authors pretrain a 1.7B parameter LLaMA-style causal decoder-only model from scratch on nine typologically diverse languages. Crucially, they save checkpoints at an exceptionally fine granularity—every 185 million tokens—capturing 200 snapshots across training.

To peer inside the model over time, they combine several advanced interpretability techniques:

  • Logit Lens: Projects intermediate hidden states from different layers directly into the vocabulary space to see what the model is "thinking" at each layer.
  • ExPLAIND: A training data and model component attribution framework used to quantify which specific parameters and training data types influence the model's predictions.
  • Causal Parameter Ablations: Artificially scaling specific weight groups (exciting them by 2.0x or inhibiting them by 0.5x) to prove their causal role in copying vs. translating.
  • Layer-Swapping Experiments: Stitching blocks from the final trained model into earlier checkpoints to isolate when and where specific capabilities form.

Datasets

  • Pretraining Data: English data from FineWeb-HQ and eight other languages (French, German, Indonesian, Italian, Portuguese, Spanish, Mandarin Chinese, Japanese) from FineWeb2-HQ.
  • Linguistic Evaluation: Subsets of MultiBLiMP, JBLiMP (Japanese), and ZhoBLiMP (Chinese) to evaluate basic syntactic capabilities like subject-verb agreement.
  • Word-Level Translation (WLT) Benchmark: A novel, curated dataset introduced by the authors spanning 72 directed language pairs. Unlike previous benchmarks that focus only on simple nouns, this dataset is balanced across parts of speech (nouns, verbs, adjectives, adverbs) and frequency ranks, accounting for polysemy and inflections.

Key Insights and Ideas

  • The Two-Phase Learning Trajectory: Translation capabilities develop in two clear stages. In Phase I (0 to ~11B tokens), the model rapidly masters basic syntax and relies heavily on surface-level token copying. In Phase II (~11B tokens onward), it transitions to true, abstract translation.
Figure 1: Training dynamics of word-level transla- tion over pretraining. The stacked areas decompose outputs into correct translations and error types; the black line tracks overall translation accuracy. Early training (Phase I) is characterized by frequent copying, whereas translation develops in later training (Phase II).
  • Copying Evolves from Chaos to Order: Early copying is untargeted "context copying" (repeating random words from few-shot prompts). Over time, it refines into targeted "source word copying" before the model learns to suppress it when inappropriate.
  • Anatomy of the Copy Mechanism: ExPLAIND and ablations revealed that attention value projections in layers 0–2 causally control whether the model copies, while layers 6–9 act as calibrators that control where to copy from.
  • The "Concept Space" Forms Early: Layer-swapping experiments showed that the intermediate layers (layers 10–15), which process abstract language-independent concepts, are largely formed by the halfway point of training (18.8B tokens). The bottleneck in later training isn't learning how to translate concepts, but rather refining how the bottom and top layers map specific languages into and out of this shared space.

Key Results

  • Rapid Syntactic Mastery: Basic linguistic acceptability (BLiMP scores) and simple repetition accuracy shoot past 90% very early on, by just 9.5 billion tokens processed.
  • The Overlap Advantage: Word pairs with full token overlap are mastered first, followed by partial overlap. True translation for words with no token overlap (e.g., across different scripts like English to Chinese) only begins to emerge around the 11-billion-token mark.
Figure 2: Evolution of model performance across diverse linguistic tasks over training.
  • Final Performance: The model achieves a final mean word-level translation accuracy of 32.1% at 37B tokens.
  • Directional Asymmetry: The model exhibits a strong bias toward translating into English rather than from English, reflecting English's dominant 50% sampling weight during pretraining.

Limitations and Future Work

  • Simplified Testbed: The study focuses strictly on word-level translation (WLT). How these dynamics scale to phrase-level translation or complex, multi-step cross-lingual reasoning remains an open question.
  • Early Termination: The 1.7B model was trained up to 37 billion tokens and did not fully converge. While this is the ideal window for studying emergent properties, fully trained models might exhibit further refinement or different late-stage dynamics.
  • Evaluation Exhaustiveness: The WLT benchmark relies on curated synonym sets; if the model generates a valid translation not captured in the dataset, it is penalized, potentially underestimating true accuracy.

Summary updated

Abstract

Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We use word-level translation as a testbed, introducing a novel dataset to trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.
BibTeX
@misc{korner2026copy,
  author = {Felicia Körner and Maria Matveev and Florian Eichin and Gitta Kutyniok and Barbara Plank and Michael A. Hedderich},
  title = {{Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining}},
  year = {2026},
  eprint = {2604.17633},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.17633},
}

Loading related content...

Figure