When a large language model (LLM) is trained on a mixture of dozens of languages, it faces an optimization challenge. It could theoretically allocate separate, redundant internal parameters to handle the grammar and facts of each language independently. Alternatively, it could compress this information into a single, language-agnostic representation space.
Recent mechanistic interpretability research reveals that models overwhelmingly prefer compression. Instead of maintaining isolated linguistic silos, multilingual transformers naturally converge on a modular, three-stage internal translation architecture. They use language-specific "gatekeepers" at their physical boundaries, but route internal reasoning through a centralized, English-biased "semantic hub."
The Outer Layers: Language-Specific Gatekeepers
A multilingual model's first challenge is handling disparate surface-level features, such as different alphabets, scripts, and tokenization patterns. To manage this without cluttering its deeper reasoning layers, the model deploys specialized subsets of parameters at its absolute boundaries.
Tang et al. identified these structures as language-specific neurons, which are heavily concentrated in the very first and very last layers of the network. These neurons act as localized translators. At the input layer, they ingest language-specific tokens and strip away their surface identity, projecting the underlying meaning into a shared internal space. At the output layer, the process reverses: language-specific neurons take an abstract concept and map it back into the vocabulary of the target language.
This boundary-level translation is highly sophisticated. For example, when processing non-Roman scripts like Hindi or Arabic, models often utilize an intermediate step termed Latent Romanization. As Saji et al. discovered, models internally map native-script tokens into a Romanized subword space within the early layers, using the Roman alphabet as a structural bridge to connect disparate scripts to a common conceptual framework.
The Middle Layers: The English Pivot and the Semantic Hub
Once an input passes through the initial gatekeepers, it enters the middle layers of the network. Here, the model does not "think" in the source language. Instead, it maps concepts into a unified Semantic Hub—a language-agnostic internal space where representations of equivalent concepts align across languages.
Crucially, this semantic hub is not entirely neutral; it is heavily anchored around English. Because English dominates the pretraining data of most large language models, the geometry of the shared representation space naturally defaults to an English-centric scaffold.
[Input: Non-English Text]
│
▼ (Layer 0: Language-Specific Neurons / Latent Romanization)
[Language-Agnostic Vector Space]
│
▼ (Middle Layers: Internal Translation to English Pivot)
[English-Centric Concept Space] ───► (Reasoning & Fact Retrieval via Universal Circuits)
│
▼ (Final Layers: Translation back to Target Language)
[Output: Target Language Text]
This "thinking in English" phenomenon has been verified by peering directly into the hidden states of running models. Wendler et al. and Gal et al. demonstrated that when a transformer processes a non-English query, it rapidly translates the intermediate vectors into an English-biased concept space in its middle layers. Zhao et al. formalized this into a distinct three-stage workflow:
- Convert the non-English input into an English-centric representation.
- Perform abstract reasoning and pull facts from memory using this English pivot.
- Translate the final concept back into the target language in the final layers.
By isolating the tiny fraction of neurons (~0.1%) responsible for this internal routing, researchers can manipulate or dramatically boost a model's performance in specific target languages.
Universal Circuits and Language-Independent Knowledge
Why does the model go through the trouble of an internal translation detour? The answer lies in parameter efficiency. By routing all languages through a central semantic hub, the model can reuse the exact same computational "circuits" to solve problems, regardless of the input language.
This reuse applies to both factual knowledge and grammatical processing:
- Factual Knowledge: Chen et al. discovered Language-Independent Knowledge Neurons (LIKNs) in the middle layers of multilingual models. When a model stores a fact (e.g., "The capital of France is Paris"), it does not store it multiple times for French, English, and Japanese. It stores it once in an abstract, language-transcendent form within these LIKNs.
- Grammatical Processing: Rather than learning separate syntactic rules for every language, models recruit universal structural circuits. Ferrando et al. showed that models use nearly identical internal attention heads to compute subject-verb agreement across vastly different languages. A "plurality direction" discovered in English activations can be used to cross-lingually force a model to generate plural verbs in Spanish or Italian, proving that the underlying algorithmic circuit is language-agnostic.
This structural alignment is so pronounced that Chi et al. found that multilingual models naturally construct overlapping syntactic subspaces for universal grammatical relations, even without ever being explicitly trained on translation or parallel data.
The Geometry of Compression
This elegant, aligned architecture is not present at the very beginning of training; it is an emergent property driven by scale and optimization.
Riemenschneider demonstrates that as language models train, they undergo a transition from maintaining language-specific representations to compressing them into shared, universal neurons. Similarly, Zeng et al. observed that as models scale up in size and training duration, their language-specific boundary regions shrink, and they rely more heavily on the shared "Lingua Franca" of their internal semantic hub.
The resulting internal geometry is highly structured. Peng shows that multilingual LLMs implicitly develop linear maps that align concepts across languages with near-isomorphic precision.
Furthermore, this internal map is not arbitrary; it preserves real-world relationships. Kudugunta et al. and Beinborn et al. revealed that the representation space naturally clusters languages by linguistic family, phylogeny, and geography. The model's internal distance between two languages reflects their actual evolutionary distance in human history.
This geometric alignment explains why cross-lingual transfer works so well. When a model learns a task in a high-resource language like English, it modifies parameters within the shared semantic hub. Because low-resource languages are mapped into that very same hub, they instantly inherit the new capability, unlocking the "magic" of zero-shot cross-lingual generalization.