Can Programming Languages Boost Each Other via Instruction Tuning?

Daoguang Zan, Ailun Yu, Bo Shen, Jiaxin Zhang, Taihong Chen, Bing Geng, Bei Chen, Jichuan Ji, Yafen Yao, Yongji Wang, Qianxiang WangarXiv 2023 · DOI · arXiv

Introduction

When a human programmer learns a new language, they don't start from scratch. Concepts like loops, variables, and logic transfer from Python to Java or C++. Recent research into Large Language Models (LLMs) suggests that these models exhibit a similar, albeit more mysterious, synergy during the instruction-tuning phase. This paper explores how training a model on a single programming language can significantly enhance its performance across a diverse multilingual spectrum.

Methodology: The CODEM Framework

Researchers developed CODEM (Code Model), a series of models based on the StarCoder architecture (7B and 15B parameters). The core experiment involved creating eight monolingual datasets—Python, JavaScript, TypeScript, C, C++, Java, Go, and HTML—each containing approximately 9,000 instruction-response pairs.

To generate these datasets, the team used GPT-3.5 to "evolve" a seed set of Python instructions into the target languages. For logic-heavy languages, they employed in-depth evolution to increase complexity. For HTML, they used in-breadth evolution, creating new instructions tailored to markup structure rather than algorithmic logic.

Cross-Language Boosting

The primary finding is that instruction tuning is not a zero-sum game. Specializing a model in one language does not cause it to "forget" others; instead, it sharpens the model's general ability to map human intent to technical code.

Table 1: Performance gains of CODEM-Python 15B over the base StarCoder model across multiple languages.

As shown in the table above, the CODEM-Python 15B model achieved a staggering 64.63% pass@1 on HumanEval-Python. More importantly, it provided massive "boosts" to other languages, such as a 17.95% absolute improvement in Java performance, despite being tuned exclusively on Python data.

The "HTML Surprise" and General Reasoning

The most unexpected result came from HTML. Despite HTML being a markup language devoid of traditional programming logic like loops or recursion, training on it boosted Java performance by 15.24%.

Figure 1: Heatmap showing the performance boost correlation between different programming languages.

The heatmap illustrates these synergies. While languages with shared syntax (like C and C++, or JS and TS) show strong mutual reinforcement, the high correlation between HTML and Java (0.88 in the boost matrix) suggests that instruction tuning serves to "unlock" pre-existing knowledge from the pre-training phase. The model already knows the syntax; instruction tuning teaches it the structure of following a prompt.

Conclusion

The cross-language boosting effect indicates that instruction tuning for code is less about learning syntax and more about mastering the "instruction-following" paradigm. This discovery opens new avenues for improving model performance in low-resource or obscure languages by leveraging high-quality instruction data from more common languages—or even from non-logical markup languages like HTML.

Summary updated

Abstract

When human programmers have mastered a programming language, it would be easier when they learn a new programming language. In this report, we focus on exploring whether programming languages can boost each other during the instruction fine-tuning phase of code large language models. We conduct extensive experiments of 8 popular programming languages (Python, JavaScript, TypeScript, C, C++, Java, Go, HTML) on StarCoder. Results demonstrate that programming languages can significantly improve each other. For example, CodeM-Python 15B trained on Python is able to increase Java by an absolute 17.95% pass@1 on HumanEval-X. More surprisingly, we found that CodeM-HTML 7B trained on the HTML corpus can improve Java by an absolute 15.24% pass@1. Our training data is released at https://github.com/NL2Code/CodeM.
BibTeX
@misc{zan2023can,
  author = {Daoguang Zan and Ailun Yu and Bo Shen and Jiaxin Zhang and Taihong Chen and Bing Geng and Bei Chen and Jichuan Ji and Yafen Yao and Yongji Wang and Qianxiang Wang},
  title = {{Can Programming Languages Boost Each Other via Instruction Tuning?}},
  year = {2023},
  eprint = {2308.16824},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.16824},
}
Figure