Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach

Zhenlan Ji, Pingchuan Ma, Zongjie Li, Shuai WangarXiv 2023 · OpenReview

Abstract

While Large Language Models (LLMs) have revolutionized code generation, their black-box nature makes evaluating and explaining their performance difficult. This paper introduces a causality-centric approach to systematically analyze the relationships between input prompts and generated code. By establishing causal graphs over fine-grained linguistic features, the researchers identify how specific prompt adjustments influence code quality, providing actionable insights for prompt engineering and model calibration.

Introduction: Beyond the Black Box

LLMs like GPT-4 have become central to modern software development, yet they remain frustratingly opaque. A minor, seemingly insignificant change in a natural language prompt can lead to a massive swing in the quality of the output—a phenomenon known as prompt instability.

Current benchmarks, such as HumanEval, are effective at measuring if a model is performing well, but they fail to explain why. They do not capture the complex interplay between the input prompt and the resulting code. Furthermore, existing explainability methods often require access to internal model weights, which is impossible for closed-source models. This research seeks to "disentangle" these relationships using a causality-centric approach, treating the LLM as a system where specific input "treatments" cause observable changes in output "effects."

The Causal Framework

The researchers developed a multi-stage pipeline to map the causal journey from prompt to code, moving from raw text to quantifiable causal relationships.

The proposed framework for causal analysis of LLM code generation.
  1. Prompt Quantification: Instead of treating a prompt as raw text, the framework extracts 255 linguistic features (lexical, syntactic, and semantic) to represent it numerically.
  2. Prompt Adjustment (Rephrasing): To observe causal effects, the researchers use an LLM to rephrase programming problems in 22 different ways (e.g., "make it more formal," "as a senior programmer," "make it shorter").
  3. Causal Discovery: Handling hundreds of variables is computationally expensive. The authors use the DiBS algorithm to build a Directed Acyclic Graph (DAG), linking rephrasing instructions to linguistic features, and those features to code metrics.
  4. Causal Inference: Using Double Machine Learning (DML), they calculate the Average Treatment Effect (ATE). This allows them to quantify how a specific change (like increasing fluency) causes a specific outcome (like a higher pass rate) while controlling for confounding variables.
  5. Downstream Optimization: The resulting causal graph acts as a "lightweight surrogate" for a genetic algorithm, searching for optimal prompt combinations to maximize code quality without expensive LLM iterations.

Key Insights and Results

The study utilized the APPS (Automated Programming Progress Standard) dataset, evaluating three representative models: GPT-Neo (2.7B), GPT-3.5-Turbo, and GPT-4.

Impact of different prompt instructions on code generation metrics.
  • The "Ignorant Idler" Analogy: The results suggest treating LLMs like an "ignorant idler." Simply making a prompt longer or more formal often hurts performance. Instead, removing redundant information and focusing on clarity yields the best results.
  • Role-Playing and Model Scale: For GPT-3.5 and GPT-4, "role-playing" (e.g., "You are a software engineer") was highly effective. However, for the smaller GPT-Neo, these same instructions were often counterproductive, as the model tended to overfit or memorize snippets rather than reason through the prompt.
  • The "Formal" Trade-off: As shown in the data, using the keyword "Formal" often complicated the programming logic for the LLM, whereas "Short" instructions tended to simplify the logic and improve the Pass Rate.
  • Predictive Accuracy: The causal graphs were remarkably accurate, achieving an $R^2$ score above 0.8 for most metrics, meaning the framework could explain 80% of the variance in the generated code's quality.

Conclusion

This causality-centric approach provides a roadmap for more precise prompt engineering by identifying specific linguistic features—such as the use of determiners—that have high causal impacts on code correctness. While the current study focused on Python and the GPT family, the framework is designed to be language-agnostic. Future work will expand these causal insights to other architectures like Llama and different programming languages such as C++ and JavaScript.

Summary updated

Abstract

While code generation has been widely used in various software development scenarios, the quality of the generated code is not guaranteed. This has been a particular concern in the era of large language models (LLMs)- based code generation, where LLMs, deemed a complex and powerful black-box model, is instructed by a high-level natural language specification, namely a prompt, to generate code. Nevertheless, effectively evaluating and explaining the code generation capability of LLMs is inherently challenging, given the complexity of LLMs and the lack of transparency. Inspired by the recent progress in causality analysis and its application in software engineering, this paper launches a causality analysis-based approach to systematically analyze the causal relations between the LLM input prompts and the generated code. To handle various technical challenges in this study, we first propose a novel causal graph-based representation of the prompt and the generated code, which is established over the fine-grained, human-understandable concepts in the input prompts. The formed causal graph is then used to identify the causal relations between the prompt and the derived code. We illustrate the insights that our framework can provide by studying over 3 popular LLMs with over 12 prompt adjustment strategies. The results of these studies illustrate the potential of our technique to provide insights into LLM effectiveness, and aid end-users in understanding predictions. Additionally, we demonstrate that our approach provides actionable insights to improve the quality of the LLM-generated code by properly calibrating the prompt.
BibTeX
@misc{ji2023benchmarking,
  author = {Zhenlan Ji and Pingchuan Ma and Zongjie Li and Shuai Wang},
  title = {{Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach}},
  year = {2023},
  eprint = {2310.06680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06680},
}
Figure