Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation

Luis Mayer, Christian Heumann, Matthias AßenmacherarXiv 2024 · arXiv

TL;DR

This paper evaluates five leading Large Language Models (LLMs)—ChatGPT, BingChat, Bard, Llama2, and Code Llama—on their ability to generate functional Python code from textual descriptions. The findings reveal that proprietary models, particularly ChatGPT, still vastly outperform open-source alternatives, even those explicitly fine-tuned for coding. However, when these models do successfully generate correct code, their solutions are remarkably efficient, frequently outperforming typical human submissions in both runtime and memory usage.

What Problem Does This Paper Address?

As LLMs become staples in software engineering workflows, a critical question remains: Can open-source models truly compete with proprietary giants like OpenAI's ChatGPT for code generation? While standard benchmarks exist, they often evaluate code in a vacuum, focusing purely on whether a script compiles or passes basic unit tests.

This paper addresses this gap by conducting a rigorous comparative study using real-world programming challenges. The authors evaluate not just whether the models can write correct code across varying difficulty levels, but also how well that code performs computationally (runtime and memory footprint) when stacked up against thousands of human submissions.

Methods

The authors designed an automated evaluation pipeline centered around the popular competitive programming platform LeetCode.

Figure 1: Schematic overview of the evaluation pipeline for the LLMs’ performance on text-to-code generation.

The methodology consists of the following steps:

  1. Prompt Engineering: Textual problem descriptions, example inputs/outputs, and constraints were extracted from LeetCode and structured into a clean prompt, appended with a explicit instruction to write the solution as a specific Python class/function.
  2. Code Generation: Prompts were fed into five state-of-the-art models: ChatGPT (GPT-3.5 turbo), BingChat (GPT-4 based), Bard (PaLM 2 based), Llama2 (70B chat), and Code Llama (34B Instruct).
  3. Post-Processing: Crucially, the authors observed that open-source models (especially Code Llama) frequently outputted code inside standard text blocks without proper Python indentation. To ensure a fair trial, a manual post-processing step was introduced to fix indentation errors before testing.
  4. LeetCode Evaluation: The code was submitted to LeetCode's live engine to verify correctness and harvest percentile rankings for execution runtime and memory usage.

Datasets

The study utilized a dataset of 89 programming tasks randomly sampled from LeetCode. To keep the evaluation focused on math- and statistics-adjacent logic, the tasks were drawn from three specific categories:

  • Math (30 tasks)
  • Matrix (30 tasks)
  • Counting (29 tasks)

Each category was evenly split across LeetCode’s three standard difficulty tiers: Easy, Medium, and Hard (with the exception of Counting, which only had 9 Hard tasks available at the time). Rather than introducing a new static benchmark, the authors leveraged LeetCode's dynamic repository of human submissions to calculate comparative performance quantiles.

Key Insights and Ideas

  • Proprietary Dominance Over Open-Source Specialization: General-purpose proprietary models comfortably beat open-source code specialists. ChatGPT (built on GPT-3.5) easily outperformed Code Llama, despite the latter being explicitly fine-tuned on massive code repositories.
  • The Indentation Trap: Code Llama consistently failed to output code with valid Python indentation when it embedded code within conversational text blocks. Without the authors' post-processing intervention, Code Llama's success rate would have dropped to near zero.
  • Context Bloat is Real: Shorter, more concise prompts were strongly correlated with successful code generation. As prompt length increased—often due to verbose problem descriptions or excessive examples—the likelihood of the model producing incorrect code rose significantly.
Figure 5: Distribution of prompt lengths of correct and incorrect solutions for BingChat and ChatGPT
  • Logic Over Syntax: The vast majority of model failures were due to "Wrong Answers" (failing edge cases in the test suites) rather than "Syntax Errors." This suggests that LLMs are generally capable of writing syntactically sound Python code but struggle to grasp complex algorithmic logic and constraints.
  • Algorithmic Efficiency: When LLMs successfully solved a problem, their code was highly optimized. The correct solutions frequently placed well above the 50th percentile of human submissions, proving that LLMs don't just write code that works—they write code that runs fast.

Key Results

  • Overall Correctness: ChatGPT led by a massive margin, correctly solving 58% of all tasks. BingChat followed at 39%, Bard at 18%, Code Llama at 9%, and Llama2 at 7%.
Figure 2: Percentage of correctly solved tasks per LMM.
  • Handling Complexity: Only the GPT-based models (ChatGPT and BingChat) successfully solved any "Hard" tasks. ChatGPT solved 44.83% of Hard problems, while Bard, Code Llama, and Llama2 failed to solve a single one (0%).
  • Execution Performance: On Medium-difficulty tasks, ChatGPT’s correct solutions achieved an average runtime ranking of 84.25%, meaning its code executed faster than roughly 84% of human submissions for those same problems.
Figure 3: Average runtime ranking (quantiles) of the generated code per LLM and difficulty level.
  • Error Distribution: Across all models, "Wrong Answer" was the most frequent error type at 54.3%, followed by "Syntax Error" at 14.6%.

Limitations and Future Work

The authors acknowledge several limitations in their study. First, it represents a static snapshot of a rapidly evolving landscape; newer iterations of these models may yield different results. Second, the evaluation scope was relatively small (89 tasks) and restricted entirely to Python within specific mathematical domains.

Crucially, the study only evaluated single-turn generation. In real-world scenarios, developers interact with LLMs conversationally, often feeding error messages back to the model to allow it to self-correct. The authors suggest that future work should focus on multi-turn debugging dialogues and expanding benchmarks to other programming languages like Java or C++.

Summary updated

Abstract

In recent years, large language models (LLMs) have emerged as powerful tools with potential applications in various fields, including software engineering. Within the scope of this research, we evaluate five different state-of-the-art LLMs - Bard, BingChat, ChatGPT, Llama2, and Code Llama - concerning their capabilities for text-to-code generation. In an empirical study, we feed prompts with textual descriptions of coding problems sourced from the programming website LeetCode to the models with the task of creating solutions in Python. Subsequently, the quality of the generated outputs is assessed using the testing functionalities of LeetCode. The results indicate large differences in performance between the investigated models. ChatGPT can handle these typical programming challenges by far the most effectively, surpassing even code-specialized models like Code Llama. To gain further insights, we measure the runtime as well as the memory usage of the generated outputs and compared them to the other code submissions on Leetcode. A detailed error analysis, encompassing a comparison of the differences concerning correct indentation and form of the generated code as well as an assignment of the incorrectly solved tasks to certain error categories allows us to obtain a more nuanced picture of the results and potential for improvement. The results also show a clear pattern of increasingly incorrect produced code when the models are facing a lot of context in the form of longer prompts.
BibTeX
@misc{mayer2024can,
  author = {Luis Mayer and Christian Heumann and Matthias Aßenmacher},
  title = {{Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation}},
  year = {2024},
  eprint = {2409.04164},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.04164},
}

Loading related content...

Figure