How Do Agents Perform Code Optimization? An Empirical Study

Huiyun Peng, Antonio Zhong, Ricardo Andrés Calvo Méndez, Kelechi G. Kalu, James C. Davis

Abstract

As AI coding agents increasingly contribute to open-source software, their ability to perform complex tasks like performance optimization remains under-scrutinized. This study presents the first empirical comparison between performance-related Pull Requests (PRs) authored by AI agents and those by humans. By analyzing 407 PRs from the AIDev dataset, the research reveals that while AI agents mirror human optimization strategies, they suffer from a significant "validation gap." AI agents frequently rely on theoretical, static reasoning rather than empirical benchmarking to justify their changes, leading to lower merge rates and potential hallucination risks. The findings suggest that for AI agents to become reliable performance engineers, they must be integrated with automated profiling and benchmarking infrastructure.

Introduction

Performance optimization is often considered one of the "final bosses" of software engineering. Unlike fixing a syntax error or implementing a basic feature, optimization requires a deep understanding of hardware constraints, algorithmic tradeoffs, and complex system behaviors. While Large Language Models (LLMs) have proven capable of generating boilerplate code, their efficacy in the nuanced domain of performance engineering is less understood.

This study addresses this gap by conducting a comparative analysis of performance-oriented Pull Requests (PRs) authored by AI agents versus those authored by human developers. The goal is to determine if AI agents identify the same bottlenecks as humans, employ similar strategies to resolve them, and—most importantly—how they validate the efficacy of their optimizations.

Methodology

The researchers employed a multi-stage analytical pipeline to ensure a rigorous comparison:

  1. Taxonomy Expansion: The study began with the SysLLMatic catalog of optimization patterns. Finding it insufficient for real-world scenarios, the authors expanded it from 43 to 59 distinct patterns, adding critical categories such as "Build/Infrastructure" and "Network/Database."
  2. Human-in-the-Loop Annotation: To label hundreds of PRs, the team used a hybrid approach. Two state-of-the-art LLMs (GPT-4 and Gemini-Pro) initially classified the code changes. In cases of disagreement, human experts (Ph.D. students) adjudicated the final classification.
  3. Statistical Analysis: The study utilized Chi-square tests to compare optimization patterns and Mann-Whitney U tests to analyze merge times and code complexity metrics.

Dataset

The study utilized the AIDev dataset, a collection of GitHub PRs authored by both agents and humans. The final sample consisted of 407 performance-related PRs (324 AI-authored and 83 human-authored), involving the reconstruction of nearly 5,000 individual commit details for fine-grained analysis.

Key Insights

The "Mirror" Effect

One of the most striking findings is that AI agents do not discover "alien" ways to optimize code. Instead, their high-level strategies—such as prioritizing memory locality, improving algorithmic efficiency, or reducing redundant computations—almost perfectly mirror human behavior. This suggests that LLMs have successfully internalized standard "best practices" from their extensive training on human-written codebases.

The Validation Gap

While the strategies are similar, the methods of proof differ wildly. Humans are "show me" developers; they typically provide benchmarks, profiling data, or execution logs to justify a change. In contrast, AI agents are "tell me" developers. They rely heavily on static reasoning, offering theoretical arguments (e.g., "This change reduces complexity from $O(n^2)$ to $O(n)$") without providing the empirical data to back it up.

This reliance on static reasoning introduces a significant hallucination risk. The authors documented instances where agents claimed massive performance gains—such as a "7400x speedup"—in PR descriptions without providing a single line of benchmarking code to verify the claim.

The "Instant Merge" Phenomenon

Interestingly, when AI-authored performance PRs are accepted, they are merged significantly faster than human PRs (a median of 0.03 hours). This suggests that when an agent's suggestion is correct, it is often an "obvious" win—a clear-cut improvement that requires little debate or deep architectural review from maintainers.

Results and Performance Metrics

  • Adoption Rates: Human PRs are merged more frequently than AI PRs (65% vs. 57%).
  • Validation Frequency: Only 45.7% of AI PRs included any form of performance validation, compared to 63.6% for humans ($p=0.007$).
  • Evidence Quality: Among the PRs that provided validation, humans used empirical benchmarks 49% of the time. AI agents relied on benchmarks only 25% of the time, favoring static reasoning in 67.2% of cases.
  • Maintainability: AI-authored optimizations do not significantly degrade code maintainability compared to human ones. However, AI agents do produce more "extreme outliers" regarding code complexity, occasionally generating highly convoluted solutions.

Conclusion and Future Outlook

The study concludes that while AI agents are competent at identifying and suggesting standard optimizations, their inability to provide empirical proof is a major barrier to adoption. The authors argue that the next generation of AI coding agents must move beyond "intuition" and static reasoning.

To become truly reliable performance engineers, AI agents must be integrated with automated profiling and benchmarking infrastructure. An agent should not just claim a speedup; it should be capable of spinning up a test environment, running the code under load, and reporting hard numbers. Until this loop is closed, the "validation gap" will remain a primary hurdle in the path toward fully autonomous software optimization.

Summary updated

BibTeX
@inproceedings{peng2026how,
  author = {Huiyun Peng and Antonio Zhong and Ricardo Andrés Calvo Méndez and Kelechi G. Kalu and James C. Davis},
  title = {{How Do Agents Perform Code Optimization? An Empirical Study}},
  booktitle = {MSR},
  year = {2026},
  doi = {10.1145/3793302.3793564},
}
Figure