Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models

Haneul Yoo, Jiho Jin, Kyunghyun Cho, Alice Oh

Abstract

While large language models (LLMs) exhibit strong multilingual abilities, their reliance on English as latent representations creates a translation barrier, where reasoning implicitly depends on internal translation into English. When this process fails, performance in non-English languages deteriorates sharply, limiting the inclusiveness of LLM-based applications. Existing cross-lingual in-context learning (X-ICL) methods primarily leverage monolingual demonstrations, often failing to mitigate this barrier and instead reinforcing it. In this work, we introduce code-switching in-context learning (CSICL), a simple yet effective prompting strategy that progressively transitions from a target language to English within demonstrations and instruction to facilitate their latent reasoning in English. By explicitly scaffolding the reasoning process through controlled code-switching, CSICL acts as an implicit linguistic bridge that enhances cross-lingual alignment and reduces reliance on the translation barrier. We conduct extensive experiments across 4 LLMs, 6 datasets, and 10 languages, spanning both knowledge-intensive and reasoning-oriented domains. Our results demonstrate that CSICL consistently outperforms X-ICL baselines, achieving gains of 3.1%p and 1.9%p in both target and unseen languages, respectively. The improvement is even more pronounced in low-resource settings, with gains of 14.7% in target and 5.3% in unseen languages. These findings establish code-switching as a principled and robust approach for overcoming the translation barrier during inference, moving LLMs toward more equitable and effective multilingual systems.

Introduction: The Translation Barrier in LLMs

Despite their impressive multilingual capabilities, modern Large Language Models (LLMs) remain notoriously English-centric. Research suggests these models often rely on a "translation barrier": they internally translate non-English queries into English latent representations to perform reasoning. When this implicit translation step fails—a common occurrence in low-resource languages or complex reasoning tasks—the model’s performance plummets.

Existing Cross-Lingual In-Context Learning (X-ICL) methods typically use either purely monolingual demonstrations (all English or all target language) or parallel pairs. However, these methods often fail to bridge the linguistic gap and may even reinforce the barrier. The challenge lies in designing an inference-time strategy that aligns non-English inputs with the model's high-performance English reasoning space without requiring additional training.

Methodology: Code-Switching In-Context Learning (CSICL)

To address this, we propose Code-Switching In-Context Learning (CSICL), a prompting strategy that uses "gradual code-switching" to nudge the model toward English-centric reasoning. The method consists of two main components:

  1. Gradual Code-Switching Demonstrations: Instead of standard few-shot examples, CSICL provides demonstrations that evolve. A single example starts 100% in the target language and, through a series of steps (e.g., 25%, 50%, 75% English), concludes as a full English equivalent. This follows the Matrix Language Frame (MLF) model, a linguistic framework for how languages interleave.
  2. Gradual Translation Instruction: The system prompt explicitly instructs the model to "gradually translate this non-English query into English, then think in English, and finally answer the question."
Figure 1: Overview of the CSICL framework, showing the gradual transition from target language to English as a few-shot hint for thinking.

To generate these high-quality, code-switched demonstrations, we developed a two-step pipeline using advanced models to convert parallel sentences into natural-sounding, progressively translated sequences.

Experimental Evaluation

We conducted an extensive evaluation across four state-of-the-art LLMs (Qwen3-32B, DeepSeek-chat-v3.1, Grok-4-fast, and Gemini 2.5 Flash) and 10 languages. The benchmarks included:

  • Global MMLU: General world knowledge across various subjects.
  • FLORES+: Machine translation tasks (specifically to English).
  • MedExpQA: Domain-specific medical knowledge.
  • PolyMath: Mathematical reasoning.
  • BLEnD: Cultural knowledge.
  • MBBQ: Social bias measurement.

The languages spanned different resource levels, including high-resource (French, Spanish, Chinese), mid-resource (Korean, Indonesian, Turkish), and low-resource (Yoruba, Swahili, Telugu).

Key Results and Analysis

CSICL consistently outperformed standard X-ICL baselines across all models and datasets. On Global MMLU, it achieved average gains of 3.1%p in target languages and 1.9%p in unseen languages.

Figure 3: Performance comparison across various models. CSICL (green line) consistently achieves the highest performance compared to monolingual and translation baselines.

Scaffolding the Latent Process

CSICL acts as an "implicit linguistic bridge." By forcing the model to externalize the translation process step-by-step, it prevents the "silent failures" that occur during hidden internal translations. One of the most striking findings was that transitioning from a target language to English yielded much better results than the reverse. This confirms the hypothesis that LLMs "think" more effectively in English; code-switching acts as a ramp into that superior reasoning space.

Performance in Low-Resource Settings

The impact was most dramatic in low-resource settings (like Yoruba), where CSICL saw improvements of up to 14.7% in target languages and 5.3% in unseen languages compared to monolingual baselines. This suggests the technique fosters a general language-agnostic alignment rather than just memorizing specific word mappings.

Conclusion and Future Directions

CSICL establishes code-switching as a principled approach for overcoming the translation barrier during inference. While effective, several areas remain for exploration. Future work could investigate intra-sentential switching (mixing languages within a single sentence) to see if finer-grained scaffolding helps. Additionally, human qualitative assessment could provide deeper insights into the "naturalness" and logic of the model's intermediate code-switched reasoning steps. Finally, finding ways to achieve similar results with more concise prompts could help mitigate the increased token usage associated with providing multiple versions of every query.

Summary updated

Abstract

While large language models (LLMs) exhibit strong multilingual abilities, their reliance on English as latent representations creates a translation barrier, where reasoning implicitly depends on internal translation into English. When this process fails, performance in non-English languages deteriorates sharply, limiting the inclusiveness of LLM-based applications. Existing cross-lingual in-context learning (X-ICL) methods primarily leverage monolingual demonstrations, often failing to mitigate this barrier and instead reinforcing it. In this work, we introduce code-switching in-context learning (CSICL), a simple yet effective prompting strategy that progressively transitions from a target language to English within demonstrations and instruction to facilitate their latent reasoning in English. By explicitly scaffolding the reasoning process through controlled code-switching, CSICL acts as an implicit linguistic bridge that enhances cross-lingual alignment and reduces reliance on the translation barrier. We conduct extensive experiments across 4 LLMs, 6 datasets, and 10 languages, spanning both knowledge-intensive and reasoning-oriented domains. Our results demonstrate that CSICL consistently outperforms X-ICL baselines, achieving gains of 3.1%p and 1.9%p in both target and unseen languages, respectively. The improvement is even more pronounced in low-resource settings, with gains of 14.7% in target and 5.3% in unseen languages. These findings establish code-switching as a principled and robust approach for overcoming the translation barrier during inference, moving LLMs toward more equitable and effective multilingual systems.
BibTeX
@inproceedings{yoo2026code,
  author = {Haneul Yoo and Jiho Jin and Kyunghyun Cho and Alice Oh},
  title = {{Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=qUilKrTry5},
}
Figure