Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs

Haichuan Hu, Ye Shang, Guowen Xu, Chao He, Quanjun Zhang2025 · DOI

TL;DR

This paper presents the first empirical study comparing the latest GPT-O1 models against previous GPT-family LLMs for automated program repair (APR). The GPT-O1 models demonstrate superior bug-fixing capabilities, successfully repairing all 40 bugs in the QuixBugs benchmark, and its "chain of thought" reasoning offers insights into fixing complex bugs.

What Problem Does This Paper Address?

Automated Program Repair (APR) is a crucial area in software quality assurance, aiming to automatically fix bugs in code. Large Language Models (LLMs), particularly OpenAI's ChatGPT, have shown remarkable effectiveness in this domain, achieving state-of-the-art performance. However, despite continuous upgrades to GPT models, there's a lack of systematic comparison across different versions regarding their APR capabilities. This paper addresses this gap by evaluating the newly released GPT-O1 models against their predecessors (GPT-40 and ChatGPT) on the QuixBugs benchmark, exploring their effectiveness, efficiency, and behavioral differences in bug fixing.

Methods

The researchers designed a two-step bug-fixing process for evaluating LLMs.

  1. Initial Repair: A basic prompt template was used to ask the LLM to identify and fix bugs in a given program.
  2. Secondary Repair: For programs that failed the initial fix, error messages from dynamic execution were provided to the LLM to aid in a more targeted repair attempt.

The evaluation focused on several dimensions:

  • Repair Success Rate: Measured by the correctness of generated patches against available test cases.
  • Thinking Time: Specifically for GPT-O1, the time spent in its "thinking" phase before generating a response was recorded.
  • Response Length: The token count of the generated repair code was analyzed to understand output verbosity and potential cost implications.
  • Model Behavior Patterns: The thought process and interaction style of the LLMs during the repair process were observed and analyzed.

Datasets

The study utilized the QuixBugs benchmark, a widely adopted dataset in the APR literature. It consists of 40 buggy Python programs, each with corresponding test cases. The paper does not introduce a new benchmark.

Key Insights and Ideas

  • GPT-O1's Leap in Performance: The GPT-O1 models (O1-preview and O1-mini) significantly outperform GPT-40 and older ChatGPT versions, achieving a perfect 40/40 bug fix rate on QuixBugs.
  • The Power of "Chain of Thought": GPT-O1's unique "chain of thought" (COT) reasoning mechanism appears crucial for tackling complex bugs. By explicitly thinking through the problem, it can better understand code logic and devise more accurate repairs, as demonstrated in the "hanoi" problem case study.
  • Trade-offs in Verbosity and Cost: GPT-O1 models produce significantly longer responses (over 50% more tokens) compared to GPT-40. While this suggests more comprehensive explanations, it also implies higher computational costs and potentially longer repair times.
  • Behavioral Differences: GPT-40 tends to provide the code fix first, then an explanation. In contrast, GPT-O1 exhibits a more human-like problem-solving approach: analyzing the problem, generating a solution, and then explaining it. This structured approach, especially the self-correction observed during the secondary repair step, highlights its advanced reasoning.

Key Results

  • Perfect Repair Rate: GPT-O1-preview and O1-mini successfully repaired all 40 bugs in the QuixBugs benchmark.
  • Improved Performance over GPT-40: Before receiving error information, O1-mini and O1-preview fixed 37 and 38 bugs, respectively, outperforming GPT-40's 35. After receiving error information, O1 models fixed all 40 bugs, while GPT-40 could only fix 38.
  • Response Length: Average token lengths were 1450 for O1-preview and 1086 for O1-mini, compared to 654 for GPT-40.
  • Thinking Time: O1-preview averaged 19.82 seconds of thinking time, while O1-mini averaged 7.02 seconds. This indicates a substantial increase in processing time for O1 models, estimated to be over 70% more than previous GPT models.

Limitations and Future Work

  • Trial Phase Limitations: GPT-O1 is currently in a trial phase with usage limits and high API costs, which restricts extensive experimentation.
  • Benchmark Size: The QuixBugs benchmark, while standard, is relatively small. The authors suggest that further research with larger datasets is needed to fully explore GPT-O1's capabilities.
  • Cost-Effectiveness: For simpler programs, traditional GPT models might be more practical due to lower time and monetary costs, despite GPT-O1's superior performance on complex tasks.
  • Future Research: The paper serves as a foundation for future in-depth exploration of GPT-family models in APR, particularly focusing on understanding and leveraging the "chain of thought" mechanism for more complex bug-fixing scenarios.

Summary updated

Abstract

LLMs have long demonstrated remarkable effectiveness in automatic program repair (APR), with OpenAI's ChatGPT being one of the most widely used models in this domain. Through continuous iterations and upgrades of GPT-family models, their performance in fixing bugs has already reached state-of-the-art levels. However, there are few works comparing the effectiveness and variations of different versions of GPT-family models on APR. In this work, inspired by the recent public release of the GPT-o1 models, we conduct the first study to compare the effectiveness of different versions of the GPT-family models in APR. We evaluate the performance of the latest version of the GPT-family models (i.e., 01-preview and 01-mini), GPT-40oo, and the historical version of ChatGPT on APR. We conduct an empirical study of the four GPT-family models against other LLMs and APR techniques on the QuixBugs benchmark from multiple evaluation perspectives, including repair success rate, repair cost, response length, and behavior patterns. The results demonstrate that 01 's repair capability exceeds that of prior GPT-family models, successfully fixing all 40 bugs in the benchmark. Our work can serve as a foundation for further in-depth exploration of the applications of GPT-family models in APR.
BibTeX
@misc{hu2025can,
  author = {Haichuan Hu and Ye Shang and Guowen Xu and Chao He and Quanjun Zhang},
  title = {{Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs}},
  year = {2025},
  doi = {10.1109/apr66717.2025.00007},
}
Figure