Internal English Pivoting in Multilingual Large Language Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2156 papers172 conceptsfrozen 2026-10-04

How does a model like Llama, trained predominantly on English text, manage to answer a complex reasoning prompt in Bengali or Swahili? The intuitive answer might be that the model has learned a "universal interlingua"—a language-agnostic representation of concepts. However, recent evidence suggests a more specific and surprising mechanism: multilingual transformers often function as internal translation engines, mapping non-English inputs into an English-centric "concept space" to perform reasoning before translating the result back into the target language.

The Sandwich Pattern of Language Processing

Research into the internal layers of transformers reveals a consistent "sandwich" structure in how they handle different languages. Weize Liu and colleagues found that Large Language Models (LLMs) utilize language-specific neurons primarily at the periphery—the very first and very last layers—while the middle layers consist of shared "multilingual experts."

This structural division serves a functional purpose. Tianyi Tang and Haoyang Huang identified these peripheral neurons as "translators." The initial layers act as an encoder that strips away language-specific surface features, while the final layers act as a decoder that re-imposes the target language's vocabulary and syntax. In between, the model operates in a space that is largely indifferent to the input language.

English as the Internal Reasoning Pivot

If the middle layers are language-neutral, what language do they "think" in? Evidence from Chris Wendler and Robert West suggests this space is not truly neutral, but heavily biased toward English. By analyzing intermediate representations, they discovered that Llama-2 typically translates non-English inputs into an English-biased representation for reasoning.

This "English pivot" explains why models often "think better" in English. Aitor Soroa and Mikel Artetxe demonstrated that a simple "self-translate" strategy—forcing the model to explicitly translate a prompt into English before solving it—consistently boosts performance. The model’s reasoning capabilities are often "locked" behind a non-English interface; once the internal state is aligned with English, the model's full logic is unleashed. Lynn Chua refers to this as the "crosslingual knowledge barrier," where knowledge learned in one language is inaccessible in another unless the model can bridge the representational gap.

The Geometry of Parallel Subspaces

Why does this internal translation work so effectively? Tyler A. Chang and Zhuowen Tu provide a geometric intuition: multilingual models organize languages into distinct but parallel linear subspaces.

Imagine two parallel planes in a high-dimensional space—one for French and one for English. While the "vocabulary" axes of these planes are different, the "structural" axes (representing syntax or semantic relationships) are nearly identical. Because these geometries are parallel, the model can "project" a concept from the French plane onto the English plane with a simple linear transformation. This allows the model to maintain the functional utility of its English-trained reasoning circuits even when the input is in a different language.

Neurons that Bridge the Gap

At the granular level, this alignment is maintained by specific types of neurons. Yuheng Chen and Pengfei Cao identified Language-Independent Knowledge Neurons (LIKNs), which store facts (like "The capital of France is Paris") in a way that triggers regardless of the input language.

Furthermore, Shimao Zhang and Shujian Huang found that the process of "alignment" (such as RLHF) actually reshapes the model's internal architecture. It pushes the model away from isolated language "silos" and toward these shared neural pathways. This explains the "blessing of multilinguality": training on a few high-resource languages can spontaneously improve performance in low-resource ones because the model is learning to navigate these shared, language-neutral circuits more efficiently.

Practical Implications for Model Steering

Understanding this internal translation workflow allows for more precise control over model behavior. Kyle Mahowald and colleagues developed AlterRep, a technique that "steers" a model's internal language identity. By pushing an embedding toward a specific language's subspace, they can force the model to switch its output vocabulary mid-sentence.

Similarly, Fei Yuan and Lei Li discovered that you don't need to retrain an entire model to add a new language. By identifying the "KS-Lottery" tickets—a tiny fraction of specific token embeddings—researchers can achieve high-quality translation by tuning as few as 18 parameters. This suggests that the "reasoning engine" is already largely language-agnostic; we only need to build the right "on-ramps" and "off-ramps" to connect new languages to the existing English-centric core.

Papers in this version

Showing 60 of 156 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 110 papers
  2. Version 1 · 2026-10-04 · 156 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_lingual_transfer_mechanisms_47f02db0,
  title = {cross-lingual transfer mechanisms},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-lingual-transfer-mechanisms/rdr_47f02db0bd044f7f84c43b984579ff6c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_english_pivoting_in_multilingua_47f02db0_v1,
  title = {Internal English Pivoting in Multilingual Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-english-pivoting-in-multilingual-large-language-models/rdr_47f02db0bd044f7f84c43b984579ff6c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}