Introduction
When a human programmer learns a new language, they don't start from scratch. Concepts like loops, variables, and logic transfer from Python to Java or C++. Recent research into Large Language Models (LLMs) suggests that these models exhibit a similar, albeit more mysterious, synergy during the instruction-tuning phase. This paper explores how training a model on a single programming language can significantly enhance its performance across a diverse multilingual spectrum.
Methodology: The CODEM Framework
Researchers developed CODEM (Code Model), a series of models based on the StarCoder architecture (7B and 15B parameters). The core experiment involved creating eight monolingual datasets—Python, JavaScript, TypeScript, C, C++, Java, Go, and HTML—each containing approximately 9,000 instruction-response pairs.
To generate these datasets, the team used GPT-3.5 to "evolve" a seed set of Python instructions into the target languages. For logic-heavy languages, they employed in-depth evolution to increase complexity. For HTML, they used in-breadth evolution, creating new instructions tailored to markup structure rather than algorithmic logic.
Cross-Language Boosting
The primary finding is that instruction tuning is not a zero-sum game. Specializing a model in one language does not cause it to "forget" others; instead, it sharpens the model's general ability to map human intent to technical code.
As shown in the table above, the CODEM-Python 15B model achieved a staggering 64.63% pass@1 on HumanEval-Python. More importantly, it provided massive "boosts" to other languages, such as a 17.95% absolute improvement in Java performance, despite being tuned exclusively on Python data.
The "HTML Surprise" and General Reasoning
The most unexpected result came from HTML. Despite HTML being a markup language devoid of traditional programming logic like loops or recursion, training on it boosted Java performance by 15.24%.
The heatmap illustrates these synergies. While languages with shared syntax (like C and C++, or JS and TS) show strong mutual reinforcement, the high correlation between HTML and Java (0.88 in the boost matrix) suggests that instruction tuning serves to "unlock" pre-existing knowledge from the pre-training phase. The model already knows the syntax; instruction tuning teaches it the structure of following a prompt.
Conclusion
The cross-language boosting effect indicates that instruction tuning for code is less about learning syntax and more about mastering the "instruction-following" paradigm. This discovery opens new avenues for improving model performance in low-resource or obscure languages by leveraging high-quality instruction data from more common languages—or even from non-logical markup languages like HTML.
Summary updated