Abstract
Large Language Models (LLMs) often struggle with low-resource languages, a phenomenon known as the "curse of multilinguality." While Cross-lingual In-Context Learning (X-ICL) can mitigate this by using a bridge language, selecting the optimal bridge remains non-trivial. This paper introduces BridgeX-ICL, a framework that leverages internal neuron activation patterns to identify effective bridge languages. By utilizing ground-truth bilingual dictionaries for accurate neuron probing and the Hilbert-Schmidt Independence Criterion (HSIC) to measure semantic dependency between activation patterns, BridgeX-ICL systematically selects bridges that enhance zero-shot performance. Experimental results on LLaMA 3 across 15 language pairs demonstrate significant improvements in Bilingual Lexicon Induction (6.02%) and Machine Reading Comprehension (5.25%), revealing that the model's internal linguistic relationships often diverge from traditional linguistic genealogies.
Introduction
The rapid advancement of Large Language Models (LLMs) has enabled impressive multilingual capabilities. However, a significant performance gap persists between high-resource languages (e.g., English) and low-resource languages (e.g., Swahili, Tagalog). This "curse of multilinguality" is often addressed via Cross-lingual In-Context Learning (X-ICL), where an intermediate "bridge" language facilitates reasoning or translation. Traditionally, bridge selection relies on heuristics like language family proximity or high-resource status. Yet, these heuristics often fail because LLMs develop their own internal representations that may not align with human linguistic history. Furthermore, existing methods for identifying "linguistic neurons" are often noisy for low-resource languages due to the model's poor initial comprehension of the input.
The BridgeX-ICL Framework
To address these challenges, we propose BridgeX-ICL, a three-stage framework for scientific bridge language selection:
- Accurate Neuron Probing: To ensure that the identified neurons are truly relevant to the target concepts, we move beyond raw text corpora. Instead, we utilize ground-truth bilingual dictionaries (e.g., MUSE) to identify word pairs that the model can already translate accurately. This targeted probing ensures that the "linguistic neurons" being analyzed are actively engaged in semantic processing.
- Identifying Overlap Neurons: We analyze the Feed-Forward Network (FFN) blocks across layers to identify neurons consistently activated across different languages. Our analysis reveals that these "overlap neurons" are concentrated in the middle layers (responsible for semantic understanding) and final layers (responsible for task-specific generation).
- HSIC-based Bridge Selection: We employ the Hilbert-Schmidt Independence Criterion (HSIC) to quantify the non-linear dependency between the activation matrices of source-target pairs and potential bridge languages. The language exhibiting the highest semantic dependency is selected as the optimal bridge.
Experimental Setup
The framework was evaluated using LLaMA 3 across 15 language pairs spanning 7 diverse language families (Indo-European, Afro-Asiatic, Austronesian, etc.). We utilized MUSE and Wiktionary_bli for dictionary-based probing. Performance was benchmarked using FLORES+ (parallel corpus), Belebele (Machine Reading Comprehension), and CLQA/XNLI (Question Answering and Natural Language Inference).
Key Insights: The Model's Internal Genealogy
Our investigation into the model's internal state yielded several critical findings:
- Internal Genealogy vs. Glottolog: The model's internal linguistic spectrum differs from human linguistic history. For instance, LLaMA 3 exhibits higher neural similarity between Arabic and French than between Arabic and Hebrew, likely reflecting training data distributions rather than genetic linguistic relationships.
- Semantic Anchoring: Visualizing latent embeddings via the "Logit Lens" shows that LLMs perform semantic reasoning in middle layers using high-resource languages like English as internal anchors, even when the input is in a low-resource language.
- Layer Importance: Middle-layer neurons are more vital for semantic transfer than final-layer neurons, which focus on the mechanics of generating specific output tokens.
Results and Discussion
BridgeX-ICL consistently outperformed zero-shot baselines. On average, it improved Bilingual Lexicon Induction (BLI) by 6.02% and Machine Reading Comprehension (MRC) by 5.25% in LLaMA 3. Notably, the method often surpassed "two-shot" learning performance (providing examples) while remaining "zero-shot" (only providing a bridge instruction), demonstrating high data efficiency. Furthermore, the optimal bridges identified for simple word translation (BLI) generalized effectively to complex reasoning tasks like reading comprehension and question answering.
Conclusion
BridgeX-ICL provides a robust, data-efficient method for enhancing LLM performance on low-resource languages by tapping into the model's internal neural patterns. By recognizing that "internal genealogy" trumps traditional linguistic proximity, we can more effectively bridge the gap in global language technology. Future work will focus on expanding benchmarks for extremely low-resource languages and exploring ways to de-bias the model's internal linguistic maps to better align with actual linguistic relationships.
Summary updated