TL;DR
This paper presents the first systematic study of cross-lingual Retrieval-Augmented Code Generation (RACG), exploring how LLMs can leverage code examples from one programming language to generate code in another. By evaluating 5 LLMs across 13 programming languages on a newly constructed 14k-instance dataset, the authors reveal that cross-lingual knowledge transfer is highly effective but unequal, heavily dependent on language affinity, and surprisingly driven by code logic rather than natural language comments.
What Problem Does This Paper Address?
Retrieval-Augmented Code Generation (RACG) has become a powerful paradigm for enhancing Large Language Models (LLMs) by retrieving relevant code snippets as external knowledge. However, existing research has almost exclusively focused on single-language settings (e.g., using Python context to generate Python code).
In the real world, software development frequently requires migrating code across different programming languages (PLs) or working with low-resource languages (like Scala) that lack extensive documentation. Because code knowledge is distributed asymmetrically across the web, developers stand to benefit immensely if an LLM can look at a solution in a high-resource language like Java or Python and translate that underlying logic to solve a problem in a completely different target language. This paper tackles the unexplored dynamics, capabilities, and limitations of cross-lingual knowledge transfer in RACG.
Methods
The authors formalize the cross-lingual RACG problem ($RACG_{l_{src} \rightarrow l_{tgt}}$) and systematically evaluate it under three distinct retrieval settings to isolate different variables:
- Golden Solution Document (Oracle Injection): The LLM is directly provided with the exact, correct solution in the source language. This establishes the theoretical upper bound of cross-lingual knowledge transfer during the generation phase.
- Top-k Retrieved Code Documents: A practical end-to-end pipeline where a retriever fetches the top-3 relevant code snippets (containing both code and natural language comments) from a source language corpus.
- Top-k Retrieved Code Documents without NL: The same end-to-end pipeline, but all natural language (NL) comments and docstrings are stripped from the retrieved snippets, leaving only pure code logic.
The study evaluates five popular LLMs, categorized into multilingual LLMs (CodeLlama-7B, Deepseek-Coder-6.7B, Qwen2.5-Coder-7B) and Python-specialized LLMs (Phi-1, Phi-1.5). For practical retrieval, they leverage CodeRankEmbed, a state-of-the-art domain-specific code retriever.
Datasets
The study utilizes two datasets:
- Small but Parallel Multi-lingual Code Dataset: Derived from HumanEval-X, containing 164 programming problems perfectly aligned across 5 PLs (Python, Java, JavaScript, C++, and Go) to ensure strict variable control during initial oracle testing.
- Large Multilingual Code Dataset: A new benchmark constructed by the authors to support broader language coverage. They extended existing datasets, utilized Qwen2-72B-Instruct to generate missing reference solutions, and rigorously verified them via unit testing. It contains 13,910 high-quality instances balanced across 13 PLs (~1,000 instances per language, including C#, Kotlin, Perl, PHP, Ruby, Scala, Swift, and TypeScript).
Key Insights and Ideas
- Linguistic Affinity Matters: Knowledge transfer is exceptionally potent between highly similar languages. For instance, JavaScript and TypeScript exhibit powerful bidirectional enhancement effects because TypeScript is a superset of JavaScript.
- Java is a Better Knowledge Vehicle than Python: Even though both languages dominate LLM pretraining datasets, retrieving Java code yields significantly higher performance gains for other target languages (+16.08% average improvement) compared to retrieving Python code (+8.29%).
- Pretraining Diversity is Mandatory: Multilingual LLMs consistently benefit from cross-lingual context. Conversely, Python-specialized LLMs struggle to comprehend non-Python context, often experiencing performance degradation because the foreign code acts as distracting interference.
- Comments are Overrated: Intuitively, one might think that natural language comments act as a semantic bridge between different programming languages. However, removing NL comments from the retrieved corpus caused only a minor drop in performance (a mere 2.37% gap for multilingual models), proving that LLMs are fully capable of transferring knowledge using pure programming logic.
Key Results
- Oracle Upper Bound: Under direct oracle injection on HumanEval-X, multilingual LLMs achieved an average Pass@1 improvement of +12.84%, while Python-specialized models only improved by +3.06%.
- End-to-End Performance: On the Large Multilingual Code Dataset, using the complete RACG pipeline with documentation yielded a +11.67% mean performance boost for multilingual LLMs.
- Retrieval Paradigms: When evaluating retrieval on pure code corpora, domain-specific retrievers (CodeRankEmbed) vastly outperformed generic text embedding models and lexical methods, achieving 91.60% Precision@5 compared to a dismal 6.57% for BM25.
Limitations and Future Work
The authors note two primary limitations:
- Model Scale: The study was restricted to relatively small models (around 7B parameters, with some supplementary 14B testing), which might not fully capture the cross-lingual capabilities of frontier-class giant models.
- Dataset Scale: While covering 13 languages, the dataset is capped at roughly 1,000 samples per language.
Future work will focus on scaling up evaluations to larger code LLMs and expanding the benchmark dataset using advanced synthetic data generation techniques.