CodeM: Less Data Yields More Versatility via Ability Matrix

Daoguang Zan, Ailun Yu, Wei Liu, Bo Shen, Shaoxin Lin, Yongshun Gong, Yafen Yao, Yan Liu, Bei Guan, Weihua Luo, Yongji Wang, Qianxiang Wang, Lizhen CuiACL 2024 · OpenReview

Abstract

In the era of code large language models (code LLMs), data engineering plays a pivotal role during the instruction fine-tuning phase. To train a versatile model, previous efforts devote tremendous efforts into crafting instruction data covering all the downstream scenarios. Nonetheless, this will incur significant expenses in constructing data and training model. Therefore, this paper introduces CodeM, a novel data construction strategy, which can efficiently train a versatile model using less data via our newly proposed ability matrix. CodeM uses ability matrix to decouple code LLMs’ abilities into two dimensions, constructing a lightweight training corpus that only covers a subset of target scenarios. Extensive experiments on HumanEvalPack and MultiPL-E imply that code LLMs can combine the single-dimensional abilities to master composed abilities, validating the effectiveness of CodeM.

Introduction

Instruction fine-tuning (IFT) is the primary method for transforming base Large Language Models (LLMs) into specialized assistants. However, in the coding domain, the sheer variety of programming languages and tasks creates a "data engineering bottleneck." Traditionally, mastering $L$ languages across $T$ tasks (e.g., generation, explanation, repair) requires $L \times T$ distinct instruction-response scenarios. This combinatorial explosion makes data construction and model training prohibitively expensive.

This paper introduces CodeM, a strategy that leverages a novel "Ability Matrix" to achieve high versatility with significantly less training data. By strategically selecting a subset of training scenarios, CodeM enables models to "fill in the blanks" through generalization and imitation.

Methodology: The Ability Matrix

The core innovation of CodeM is the decoupling of model abilities into two dimensions: Languages and Tasks. Instead of filling every cell in the $L \times T$ grid, CodeM identifies a minimal spanning subset.

The Ability Matrix: CodeM (subset) vs. SFT-Mixed (full coverage)

As illustrated in Figure 1, while the standard "SFT-Mixed" approach requires data for all 18 scenarios (in a 6x3 grid), CodeM focuses on a subset: typically one full row (all tasks for an "anchor" language like Python) and one full column (a single task across all target languages). This reduces the required scenarios to $L + T - 1$. The model masters the remaining "empty" cells through two hypothesized mechanisms:

  • Generalization: Applying a known task (e.g., Code Generation) to a new language.
  • Imitation: Learning the relationship between tasks (e.g., how "Repair" relates to "Generation") in one language and imitating that logic in another.

Experimental Results

The authors evaluated CodeM using datasets generated by GPT-3.5-turbo across benchmarks like HumanEvalPack and MultiPL-E. The results consistently showed that CodeM variants, using roughly half the data of the baseline, performed on par with or better than the full-mix approach.

Impact of data scale and noise on model performance

Efficiency and Convergence

As shown in Figure 4(a), CodeM variants (e.g., CodeM-CE#Java) reach peak performance much earlier in the training process than the SFT-Mixed baseline. In some instances, CodeM achieved an absolute 2.4% improvement in pass@1 over the full-data baseline despite using significantly fewer data points (4.3K vs 9.6K).

Robustness to Noise

Figure 4(b) demonstrates that CodeM is more resilient to data noise. When the training set was corrupted with inconsistent instructions, CodeM maintained a higher performance floor compared to the SFT-Mixed baseline, suggesting that the strategic selection of data helps the model learn more robust underlying concepts.

Cross-Domain Synergy

Interestingly, the research found that training on non-coding logic, such as math (GSM8K) or natural language QA (WizardLM), actually improved code generation performance. This suggests that "reasoning" is a shared underlying ability that transcends specific programming languages or even the coding domain itself.

Conclusion

CodeM demonstrates that in the realm of instruction fine-tuning, strategic data selection is more important than sheer volume. By utilizing the Ability Matrix, researchers can build versatile code LLMs with a fraction of the data and computational cost. Future work will investigate expanding this matrix to include more complex dimensions like data structures and specific software domains.

Summary updated

Abstract

In the era of code large language models (code LLMs), data engineering plays a pivotal role during the instruction fine-tuning phase. To train a versatile model, previous efforts devote tremendous efforts into crafting instruction data covering all the downstream scenarios. Nonetheless, this will incur significant expenses in constructing data and training model. Therefore, this paper introduces CodeM, a novel data construction strategy, which can efficiently train a versatile model using less data via our newly proposed ability matrix. CodeM uses ability matrix to decouple code LLMs’ abilities into two dimensions, constructing a lightweight training corpus that only covers a subset of target scenarios. Extensive experiments on HumanEvalPack and MultiPL-E imply that code LLMs can combine the single-dimensional abilities to master composed abilities, validating the effectiveness of CodeM.
BibTeX
@inproceedings{zan2024codem,
  author = {Daoguang Zan and Ailun Yu and Wei Liu and Bo Shen and Shaoxin Lin and Yongshun Gong and Yafen Yao and Yan Liu and Bei Guan and Weihua Luo and Yongji Wang and Qianxiang Wang and Lizhen Cui},
  title = {{CodeM: Less Data Yields More Versatility via Ability Matrix}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=o0OmPQICH8},
}
Figure