How Do Agents Perform Code Optimization? An Empirical Study

Huiyun Peng, Antonio Zhong Qiu, Ricardo Andrés Calvo Méndez, Kelechi G. Kalu, James C. Davis

TL;DR

AI coding agents are increasingly contributing performance optimizations to open-source software, but they tend to "talk the talk" more than "walk the walk." While agents use similar optimization patterns to humans—focusing heavily on memory locality and algorithmic efficiency—they are significantly less likely to provide empirical benchmarks, often relying on logical reasoning that can occasionally veer into hallucination.

What Problem Does This Paper Address?

Performance engineering is a "black art" of software development. Unlike fixing a syntax bug, making code faster requires a deep understanding of hardware, workloads, and system bottlenecks. As AI agents (like those powered by LLMs) begin submitting Pull Requests (PRs) to improve performance, a critical question arises: Are these agents actually doing "real" performance engineering, or are they just applying surface-level patterns?

The authors address this by conducting the first empirical study comparing agent-authored and human-authored performance PRs. They investigate whether AI agents choose the same optimization strategies as humans and, more importantly, how they prove their changes actually work.

Methods

The researchers employed a multi-stage pipeline to analyze real-world GitHub data:

  1. Categorization: They adopted and expanded the SysLLMatic catalog, a taxonomy of performance optimization patterns. They added 14 new patterns to better capture real-world diversity, such as infrastructure and database optimizations.
  2. Annotation: To label hundreds of PRs, they used a "human-in-the-loop" approach with two state-of-the-art LLMs (GPT-5.1 and Gemini-3-Pro) acting as initial annotators, followed by manual adjudication by Ph.D. students to ensure accuracy.
  3. Statistical Analysis: They used various statistical tests (Chi-square, Mann-Whitney U) to determine if the differences between AI and human behavior were statistically significant or just due to chance.

Datasets

The study draws from the AIDev dataset, which contains GitHub PRs authored by both agents and humans.

  • Initial Pool: Over 40,000 PRs.
  • Refined Dataset: 407 performance-related PRs (324 AI-authored, 83 human-authored), comprising nearly 5,000 individual commit records.
  • Artifacts: The authors contributed an extended performance optimization catalog and a "Perf DB" of labeled patches.

Key Insights and Ideas

  • The "Logic vs. Evidence" Gap: This is the paper's most striking insight. Humans tend to justify performance gains with benchmarks (actual measurements). AI agents, however, rely on static reasoning—explaining why the code should be faster based on logic (e.g., "reducing O(n²) to O(n)"). While logically sound, this lacks empirical grounding and is prone to error.
  • Pattern Parity: Surprisingly, AI agents aren't "hallucinating" wild new ways to optimize code. They emphasize the same high-level categories as humans: Memory/Data Locality and Algorithm-Level Optimizations. This suggests LLMs have successfully internalized standard performance engineering knowledge.
  • The Speed Paradox: Agent-authored PRs that get merged do so incredibly fast (median time: 0.03 hours vs. 2.65 hours for humans). However, they are merged less frequently overall (57% vs. 65%). This suggests that while agents are efficient at the "submission-to-merge" pipeline, their contributions face a higher skepticism barrier or lower quality threshold.
  • The Hallucination Risk: The authors found instances where agents claimed massive speedups (e.g., "7400x faster") in the PR description without providing any supporting benchmark code or data, highlighting a "trust but verify" crisis for maintainers.

Key Results

  • Validation Frequency: Only 45.7% of AI PRs included explicit performance validation, compared to 63.6% of human PRs ($p=0.007$).
  • Validation Style: Among those that did validate, humans used benchmarks 49% of the time, while agents used them only 25% of the time. Agents preferred static reasoning (67.2%).
  • Maintainability: There was no significant difference in the impact on code complexity (Cyclomatic Complexity) or size between AI and human patches. AI agents aren't necessarily making the code "messier" to make it faster.
  • Loop Optimizations: Despite decades of academic focus on loop unrolling and fusion, these patterns were almost entirely absent in both human and AI PRs, likely due to concerns about readability and compiler-level optimizations already handling them.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Self-Reporting Bias: The study only analyzes what is written in the PR. Humans might perform benchmarks locally and not report them, leading to an underestimation of human validation.
  • Dataset Asymmetry: The AIDev dataset uses different repository star-count filters for AI vs. human PRs, which could introduce bias in the types of projects analyzed.

Future Work: The authors suggest that for AI agents to become "next-gen" optimizers, they must be tightly integrated with automated profiling and benchmarking pipelines. Instead of just guessing that a change is faster, future agents should be required to provide a "performance certificate" generated by running the code in a sandboxed CI/CD environment.

Summary updated

BibTeX
@inproceedings{peng2026how,
  author = {Huiyun Peng and Antonio Zhong Qiu and Ricardo Andrés Calvo Méndez and Kelechi G. Kalu and James C. Davis},
  title = {{How Do Agents Perform Code Optimization? An Empirical Study}},
  booktitle = {MSR},
  year = {2026},
  doi = {10.1145/3793302.3793564},
}

Loading related content...

Figure