Researchers at the Japan Advanced Institute of Science and Technology (JAIST) and RIKEN have recently proposed a novel framework for understanding how Multilingual Large Language Models (LLMs) navigate between diverse languages. Their work introduces the Transfer Neurons Hypothesis, which identifies specific neurons within the Multi-Layer Perceptron (MLP) modules as the primary drivers for shifting data between language-specific representations and a shared, English-centric "universal" semantic space.
The Challenge of Multilingual Latent Spaces
While models like LLaMA and Mistral demonstrate remarkable proficiency across dozens of languages, the internal mechanics of this capability have remained largely opaque. Previous research suggested a "sandwich" architecture: early layers process language-specific inputs, middle layers perform language-agnostic reasoning, and final layers translate that reasoning back into the target language. However, the specific circuits responsible for moving data between these distinct "latent spaces" were unknown. Identifying these mechanisms is critical for improving cross-lingual transfer, particularly for low-resource languages that struggle to align with the model's internal "lingua franca."
Methodology: Identifying the "Shifters"
The authors utilized a combination of geometric analysis and targeted interventions to validate their hypothesis:
- Latent Space Visualization: Using Principal Component Analysis (PCA) and Singular Value Decomposition (SVD), the team mapped the "trajectories" of hidden states for parallel sentences across model layers.
- Transfer Neuron Scoring: They developed a metric to identify neurons in the MLP modules that, when activated, move a representation closer to the mathematical center (centroid) of the target latent space.
- Ablation Studies: To confirm functionality, they "deactivated" these neurons during inference, setting their activations to zero to measure the impact on model performance and language consistency.
- Comparative Analysis: The study compared these "Transfer Neurons" (TNs) with previously identified "language-specific neurons" to determine if they served the same functional purpose.
Key Insights: The "U-Turn" and the English Hub
The research revealed several critical insights into the internal architecture of LLMs:
- The "U-Turn" Trajectory: Representations begin in distinct language-specific clusters, converge into a shared space in the middle layers (typically layers 10–20 in a 32-layer model), and then diverge back into language-specific spaces in the final layers.
- English as the Internal Hub: The shared semantic space is not language-neutral; it is heavily English-centric. Languages linguistically distant from English, such as Japanese or Korean, require more significant "shifting" by the model than languages like Dutch or Italian.
- Functional Specialization: The authors identified two distinct types of Transfer Neurons. Type-1 TNs are located in early-to-middle layers and "upload" language-specific inputs into the shared semantic space. Type-2 TNs are found in the final layers and "download" reasoned concepts back into the specific target language.
Experimental Results
The study's findings demonstrate that Transfer Neurons are not merely correlational but functionally essential:
- Surgical Impact: Deactivating just the top 1,000 Type-1 TNs—representing approximately 0.2% of the model's total neurons—caused a catastrophic drop in reasoning accuracy. For example, on the MKQA task, LLaMA3-8B’s F1 score dropped by 0.41 for Dutch and 0.15 for Japanese.
- Language Leakage: When Type-2 TNs (the "download" neurons) were deactivated, the models often maintained correct reasoning but outputted the answer in the wrong language (typically English) or produced "gibberish" consisting of mixed scripts.
- Linguistic Proximity: The overlap of Transfer Neurons is significantly higher for languages within the same family, suggesting that the model utilizes similar circuits for linguistically related tasks.
Implications and Future Directions
While the study currently assumes a linear contribution from MLPs (parallel shifts in the residual stream), it provides a powerful new lens for model interpretability. The discovery of Transfer Neurons suggests that researchers could improve a model's multilingualism through selective fine-tuning—specifically aligning the Transfer Neurons of a new or low-resource language to the existing English-centric semantic hub, rather than retraining the entire model.
Future work will involve extending this analysis to a broader array of low-resource languages and exploring the non-linear transformations that may occur within the MLP modules.
Summary updated