From Effectiveness to Efficiency: Comparative Evaluation of Code Generated by LCGMs for Bilingual Programming Questions

Weipeng Jiang, Xuanqi Gao, Juan Zhai, Shiqing Ma, Xiaoyu Zhang, Chao Shen

Abstract

Large Code Generation Models (LCGMs) have garnered significant attention and achieved promising results across various programming tasks. However, concerns arise regarding performance when using non-English prompts, as these models are primarily trained on English-centric corpora, and most programming language tokens resemble English. Existing benchmarks often rely on English programming questions and limited manual unit test cases, inadequately assessing LCGM-generated code quality. This paper investigates code quality differences, specifically effectiveness and efficiency, when employing different natural languages as inputs, focusing on Chinese and English due to their prominent corpora and LCGM availability. Evaluating LCGM-generated code quality under bilingual inputs presents three challenges: (1) lack of high-quality bilingual programming question datasets, (2) insufficient unit test cases for comprehensive correctness verification, and (3) limited support for comparing generated code performance. To address these challenges, we curated a test suite of 52 bilingual programming questions and developed automated input generators for each. We enhanced correctness verification by sampling larger unit test cases and estimated code performance by profiling execution time relative to input size growth. Using this framework, we conducted an empirical study on six state-of-the-art LCGMs. The results revealed that LCGM-generated code exhibits varying bilingual correctness on an average of 10.5% of tasks, with 39.5% of correct code showing diverse bilingual performance differences. Our findings suggested LCGMs may not consistently generate high-quality code across different languages, providing insights for future research directions.

Introduction

As Large Language Models (LLMs) become the backbone of modern software engineering, there is a growing assumption that these tools provide an "equalizing" force for developers worldwide. However, most evaluation benchmarks remain English-centric. If a model generates an optimized $O(n \log n)$ solution when asked in English but a sluggish $O(n^2)$ solution when asked in Chinese, it creates a hidden "efficiency tax" for non-English speakers.

This research addresses the challenge of measuring linguistic bias in code generation. The authors argue that simply checking if code "works" (correctness) is insufficient; we must also measure "how well" it works (efficiency) across different languages to ensure equitable access to high-quality AI-assisted programming.

Methodology

The researchers developed a unified evaluation framework that transcends standard unit testing through four key components:

  1. Correctness Verification: An automated test input generator creates large-scale, randomized test cases to ensure generated code is robust beyond hard-coded examples.
  2. Efficiency Quantification via Curve Fitting: Rather than raw execution time, the framework estimates algorithmic complexity. By running code against varying input sizes ($n$) and using least-squares regression, the system fits data to seven complexity classes (e.g., Linear, Quadratic, Exponential).
  3. Scoring System: Models receive a score from 1 to 7 based on the efficiency of the achieved complexity class, allowing for environment-independent comparison.
  4. Interpretability Analysis: Perturbation-based techniques calculate "importance scores" for specific prompt keywords (like "efficient") to determine how much attention models pay to instructions across different languages.

Dataset and Experimental Setup

The study utilizes a newly curated dataset of 52 Python programming questions adapted from HumanEval, MBPP, and LeetCode. Questions were selected based on moderate difficulty, the potential for multiple solutions with varying complexities, and the involvement of variable-sized inputs. Each question features manually reviewed, parallel descriptions in both English and Chinese to ensure high-quality bilingual pairs.

Key Insights: The Linguistic Bias

  • The Efficiency Tax: Even when models solve a problem in both languages, they often choose less efficient algorithms for one. In 39% of cases where code was correct in both languages, the efficiency differed.
  • Temperature-Induced Bias: Bias is not static; at lower temperatures ($t=0.2$), models performed better on Chinese prompts, while at higher temperatures ($t=0.8$), the advantage shifted toward English.
  • The Prompting Paradox: Adding instructions like "be concise and efficient" (Quality-aware prompting) occasionally decreased the model's ability to generate correct code.
  • Attention Disparity: Interpretability tools revealed that models like DeepSeek-Coder paid significantly less attention to the word "efficient" in Chinese prompts compared to English ones.

Empirical Results

The study found that models showed different correctness results on average for 12% of tasks, while 39% of correctly solved problems exhibited significant efficiency disparities.

Comparison of correctness and efficiency disparities between English and Chinese prompts.

As illustrated in Figure 4, linguistic bias manifests in both logic and performance. In the LeetCode example, the English prompt leads to an erroneous implementation due to improper integer division, whereas the Chinese prompt yields the correct rounded result. Conversely, in the HumanEval example, while both versions are correct, the English prompt produces a more efficient $O(n \log n)$ algorithm compared to the $O(n^2)$ solution generated from the Chinese prompt.

Commercial models like GPT-4 demonstrated the highest overall capabilities and lowest linguistic bias, though they were not immune. Notably, scaling does not solve the issue: larger models like CodeLlama-34B sometimes showed more pronounced bias than smaller counterparts, suggesting that parameter growth does not automatically lead to linguistic equity.

Conclusion and Future Work

The research highlights that LCGMs do not consistently generate high-quality code across different languages. Current limitations include the dataset size (52 tasks) and the focus on Python and the English-Chinese pair. Future work should expand to more diverse programming languages (like Java or C++) and lower-resource natural languages. The authors call for the software engineering community to move beyond English-only benchmarks to ensure the AI coding revolution is truly global and equitable.

Summary updated

Abstract

Large Code Generation Models (LCGMs) have garnered significant attention and achieved promising results across various programming tasks. However, concerns arise regarding performance when using non-English prompts, as these models are primarily trained on English-centric corpora, and most programming language tokens resemble English. Existing benchmarks often rely on English programming questions and limited manual unit test cases, inadequately assessing LCGM-generated code quality. This paper investigates code quality differences, specifically effectiveness and efficiency, when employing different natural languages as inputs, focusing on Chinese and English due to their prominent corpora and LCGM availability. Evaluating LCGM-generated code quality under bilingual inputs presents three challenges: (1) lack of high-quality bilingual programming question datasets, (2) insufficient unit test cases for comprehensive correctness verification, and (3) limited support for comparing generated code performance. To address these challenges, we curated a test suite of 52 bilingual programming questions and developed automated input generators for each. We enhanced correctness verification by sampling larger unit test cases and estimated code performance by profiling execution time relative to input size growth. Using this framework, we conducted an empirical study on six state-of-the-art LCGMs. The results revealed that LCGM-generated code exhibits varying bilingual correctness on an average of 10.5% of tasks, with 39.5% of correct code showing diverse bilingual performance differences. Our findings suggested LCGMs may not consistently generate high-quality code across different languages, providing insights for future research directions.
BibTeX
@misc{jiang2024from,
  author = {Weipeng Jiang and Xuanqi Gao and Juan Zhai and Shiqing Ma and Xiaoyu Zhang and Chao Shen},
  title = {{From Effectiveness to Efficiency: Comparative Evaluation of Code Generated by LCGMs for Bilingual Programming Questions}},
  year = {2024},
  eprint = {2406.00602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.00602},
}
Figure