From Effectiveness to Efficiency: Uncovering Linguistic Bias in Large Language Model-based Code Generation

Weipeng Jiang, Xuanqi Gao, Juan Zhai, Shiqing Ma, Xiaoyu Zhang, Chao Shen, Shen, Chao

Abstract

Large Language Models (LLMs) have demonstrated promising capabilities for code generation. While existing benchmarks evaluate the correctness and efficiency of LLM-generated code, the potential linguistic bias—where code quality varies based on the natural language used to describe programming tasks—remains underexplored. In this paper, we aim to investigate this linguistic bias through the lens of English and Chinese. To facilitate our investigation, we present a unified evaluation framework comprising a curated dataset of 52 Python programming questions with parallel bilingual task descriptions, automated correctness verification, and efficiency quantification tools based on runtime complexity estimation. Based on this framework, we conduct the first empirical study towards the linguistic bias in LLM-generated code on eight popular LCGMs, as well as GPT-3.5-Turbo and GPT-4. We observe that these LCGM-generated code show different correctness on an average of 12% bilingual programming tasks, where 39% also exhibits diverse efficiency. Our findings indicate that LLMs commonly exhibit linguistic bias for code generation.

Introduction

As Large Language Models (LLMs) become the primary assistants for developers worldwide, a critical question arises: is the "intelligence" of these models distributed equitably across different languages? Most code generation benchmarks, such as HumanEval, are English-centric and focus almost exclusively on correctness—whether the code passes a set of unit tests. However, this overlooks the potential for linguistic bias, where a model's performance might fluctuate significantly depending on the language of the prompt.

This research addresses two major facets of this bias:

  1. Correctness Disparity: A model might solve a problem when asked in English but fail when asked in Chinese, or vice versa.
  2. Efficiency Disparity: Even if the code is functionally correct in both languages, the model might generate a highly optimized algorithm for one language and a slow, naive one for the other. This "hidden" bias can lead to higher computational costs and degraded software quality for non-English speakers.

Methodology and Evaluation Framework

To measure these disparities, the researchers developed a unified evaluation framework designed to quantify both the correctness and the algorithmic efficiency of LLM-generated code.

Performance Estimation via Complexity Curve Fitting

A key technical contribution of this work is the method for hardware-independent efficiency quantification. Instead of relying on raw execution time, which is subject to environmental noise, the framework uses Performance Estimation via Complexity Curve Fitting. The process involves:

  • Automated Input Generation: Creating test inputs of varying sizes ($n$) to observe how execution time scales.
  • Regression Analysis: Using least-squares regression to fit execution data to common complexity classes (e.g., Constant, Linear, Log-linear, Quadratic).
  • Efficiency Scoring: Assigning an "Efficiency Score" (ranging from 1 to 7) based on the resulting Big-O complexity, allowing for a fair comparison of algorithmic "intelligence."

The Bilingual Dataset

The study utilized a curated dataset of 52 Python programming questions derived from HumanEval, MBPP, and LeetCode. Each task features high-quality, manually reviewed parallel descriptions in both English and Chinese. The evaluation covered 10 models, including eight open-source Large Code Generation Models (LCGMs) like CodeLlama, DeepSeek-Coder, and StarCoder, alongside commercial models GPT-3.5-Turbo and GPT-4.

Empirical Findings

The study revealed significant linguistic bias across all tested models. On average, models showed different correctness results on 12% of bilingual tasks. More strikingly, among the problems solved correctly in both languages, 39% exhibited significant disparities in time complexity.

Examples of linguistic bias in LLM-generated code, showing disparities in both correctness and algorithmic efficiency between English and Chinese prompts.

As illustrated in the figure above, the bias manifests in two ways. In the LeetCode/1491 example, the English prompt results in an erroneous implementation (using integer division // instead of rounding), while the Chinese prompt yields the correct implementation. Conversely, in the HumanEval/0 example, both implementations are correct, but the English prompt triggers an $O(n \log n)$ sorting-based approach, while the Chinese prompt results in a nested loop with $O(n^2)$ complexity.

Factors Influencing Bias

  • Temperature Effects: At lower temperatures ($t=0.2$), models generally performed better on Chinese prompts. However, at higher temperatures ($t=0.8$), the advantage often flipped toward English.
  • Prompting Strategies: Neither "Quality-aware" prompts (asking for efficiency) nor Chain-of-Thought (CoT) prompting consistently reduced bias. In several instances, these techniques widened the gap by improving English performance while Chinese performance remained stagnant.
  • Attention Disparity: Interpretability analysis on models like DeepSeek-Coder showed that the models paid significantly less attention to key terms like "efficient" when they appeared in Chinese prompts compared to their English counterparts.

Conclusion

The findings demonstrate that LLMs do not just process logic; they are deeply influenced by the natural language of the task description. While larger models like GPT-4 show higher overall correctness, they are not immune to efficiency disparities. Furthermore, increasing model size does not inherently reduce bias; for the CodeLlama family, bias actually increased as the model scaled from 13B to 34B parameters. This research highlights the need for the AI community to develop training and alignment techniques that ensure "algorithmic equity," providing high-quality, efficient code generation for all developers regardless of their native language.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated promising capabilities for code generation. While existing benchmarks evaluate the correctness and efficiency of LLM-generated code, the potential linguistic bias - where code quality varies based on the natural language used to describe programming tasks - remains underexplored. In this paper, we aim to investigate this linguistic bias through the lens of English and Chinese. To facilitate our investigation, we present a unified evaluation framework comprising a curated dataset of 52 Python programming questions with parallel bilingual task descriptions, automated correctness verification, and efficiency quantification tools based on runtime complexity estimation. Based on this framework, we conduct the first empirical study towards the linguistic bias in LLM-generated code on eight popular LCGMs, as well as GPT-3.5-Turbo and GPT-4. We observe that these LCGM-generated code show different correctness on an average of 12% bilingual programming tasks, where 39% also exhibits diverse efficiency. Our findings indicate that LLMs commonly exhibit linguistic bias for code generation.
BibTeX
@misc{jiang2024from,
  author = {Weipeng Jiang and Xuanqi Gao and Juan Zhai and Shiqing Ma and Xiaoyu Zhang and Chao Shen and Shen, Chao},
  title = {{From Effectiveness to Efficiency: Uncovering Linguistic Bias in Large Language Model-based Code Generation}},
  year = {2024},
  eprint = {2406.00602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.00602},
}
Figure