Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages

Yuemei Xu, Kexin Xu, Jian Zhou, Ling Hu, Lin Gui

TL;DR

BridgeX-ICL is a novel method that boosts Large Language Model (LLM) performance on low-resource languages by identifying the optimal "bridge" language through internal neuron analysis. By measuring how many "linguistic neurons" a bridge language shares with a target language, the researchers can significantly improve cross-lingual transfer without the need for expensive fine-tuning.

What Problem Does This Paper Address?

While LLMs like LLaMA 3 show impressive multilingual capabilities, they still struggle significantly with low-resource languages (e.g., Swahili, Tagalog, or Hebrew). A common workaround is Cross-lingual In-Context Learning (X-ICL), where a "bridge" language (usually English) is used to help the model process a task. For example, to translate Arabic to Hebrew, one might prompt the model to translate Arabic to English first.

However, two major gaps exist:

  1. Bridge Selection: Is English always the best bridge? Human linguistic genealogy (language families) suggests closer relatives might be better, but models don't always "think" like humans.
  2. Neuron Reliability: Previous research into "language neurons" often relied on noisy data (like Wikipedia), leading to inaccurate conclusions about which neurons actually handle specific languages, especially for low-resource ones.

Methods

The authors propose BridgeX-ICL, a three-step framework:

  1. High-Precision Probing: Instead of using raw web text, they use ground-truth bilingual dictionaries (MUSE) to create word pairs. They prompt the LLM to translate these pairs bidirectionally, ensuring the specific neurons associated with those languages are "fully activated" and accurately identified.
  2. Identifying Overlap Neurons: They focus on the Feed-Forward Network (FFN) layers of the transformer. They define "overlap neurons" as those that fire frequently for both the source/target language and a potential bridge language.
  3. HSIC-based Bridge Selection: They use the Hilbert-Schmidt Independence Criterion (HSIC) to quantify the non-linear dependency between the activation patterns of different languages. This creates an "internal linguistic spectrum" for the model. The bridge language with the highest dependency score in the middle layers (where semantic understanding happens) is selected as the optimal bridge.

Datasets

The study evaluates 15 language pairs across 7 diverse families (Indo-European, Afro-Asiatic, Austronesian, etc.).

  • Tasks: Bilingual Lexicon Induction (BLI), Machine Reading Comprehension (MRC via the Belebele dataset), Question Answering (CLQA), and Natural Language Inference (XNLI).
  • Probing Data: MUSE bilingual dictionaries and the FLORES+ parallel corpus.
  • Models: LLaMA-3-8B and Mistral-7B-Instruct-v0.3.

Key Insights and Ideas

  • The Model's "Internal Taxonomy": LLMs build their own understanding of language relationships that doesn't always match human linguistics. For example, LLaMA 3 shows a stronger neural similarity between Arabic and French than between Arabic and its "relative" Hebrew. This is likely due to the distribution of languages in the training data.
  • Middle Layers are Key: The researchers found that neurons in the middle layers handle semantic understanding, while final layers are more about "decoding" or predicting the next token. Therefore, bridge selection should be based on middle-layer overlap.
  • Difficulty Triggers Activity: Low-resource languages actually trigger higher neuron activation frequencies. The authors hypothesize that the model "works harder" (activates more neurons) to process languages it is less familiar with.
  • Shared Semantics: Overlapping neurons cluster together in semantic space (visualized via UMAP), proving they encode genuine shared meaning across different language families.

Key Results

  • Performance Gains: BridgeX-ICL improved performance on the BLI task by an average of 6.02% for LLaMA 3 and 4.48% for Mistral compared to zero-shot baselines.
  • MRC Success: On the Machine Reading Comprehension task, the method provided a 5.25% average accuracy boost.
  • Optimal Bridges: While English was the best bridge in 9 out of 15 cases for LLaMA 3, other languages like Portuguese, Spanish, and even Chinese showed strong potential as bridges for specific language pairs, outperforming the standard "English-only" approach.

Limitations and Future Work

  • Benchmark Scarcity: The study was limited to 15 language pairs because high-quality evaluation benchmarks for truly low-resource languages are still rare.
  • Training Bias: The "internal linguistic spectrum" is heavily influenced by the model's pre-training data. If a model is overwhelmingly trained on English, it will naturally "prefer" English as a bridge.
  • Future Directions: The authors suggest exploring how to mitigate these training biases and investigating if these "overlap patterns" can be used to prune or compress multilingual models without losing low-resource performance.

Summary updated

Abstract

The current Large Language Models (LLMs) face significant challenges in improving their performance on low-resource languages and urgently need data-efficient methods without costly fine-tuning.From the perspective of language-bridge, we propose a simple yet effective method, namely BridgeX-ICL, to improve the zero-shot Cross-lingual In-Context Learning (X-ICL) for low-resource languages.Unlike existing works focusing on language-specific neurons, BridgeX-ICL explores whether sharing neurons can improve cross-lingual performance in LLMs.We construct neuron probe data from the groundtruth MUSE bilingual dictionaries, and define a subset of language overlap neurons accordingly to ensure full activation of these anchored neurons.Subsequently, we propose an HSIC-based metric to quantify LLMs' internal linguistic spectrum based on overlapping neurons, guiding optimal bridge selection.The experiments conducted on 4 crosslingual tasks and 15 language pairs from 7 diverse families, covering both high-low and moderate-low pairs, validate the effectiveness of BridgeX-ICL and offer empirical insights into the underlying multilingual mechanisms of LLMs.The code is publicly available at https://github.com/ xuyuemei/BridgeX-ICL.
BibTeX
@misc{xu2025linguistic,
  author = {Yuemei Xu and Kexin Xu and Jian Zhou and Ling Hu and Lin Gui},
  title = {{Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages}},
  year = {2025},
  eprint = {2508.17078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.17078},
}
Figure