Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and Maintenance

Yinuo Liu, Yunlong Lyu, Zichao Luo, Tian Dong, Zian Luo, Peng Chen, Hao ChenarXiv 2026 · arXiv

TL;DR

This paper presents the first large-scale study of over 120,000 bug reports to track how AI is shifting from a source of "automated slop" to a sophisticated tool for bug discovery and diagnosis. While AI-involved reports are becoming more complete and valid over time, the study reveals a "context gap": maintainers still reject nearly 24% of AI-generated reproducers, highlighting a persistent need for human accountability and verification.

What Problem Does This Paper Address?

In early 2024, the open-source community faced an "AI slop" crisis. Maintainers of major projects like curl and Log4j reported being overwhelmed by a surge of low-quality, AI-generated bug reports that looked credible but were technically hollow. This led some organizations to suspend bug bounty programs entirely.

However, as Large Language Models (LLMs) have improved, the narrative has begun to shift. Some maintainers now report "real" findings discovered by AI. The authors of this paper set out to move beyond anecdotes and systematically answer: Is AI-involved bug reporting actually getting better? Which AI tasks (e.g., writing vs. diagnosis) lead to valid fixes? And how are maintainers changing their workflows to handle this new reality?

Figure 1. Evolving challenges of AI-generated reports.

Methods

The researchers developed a comprehensive measurement framework to trace the lifecycle of a bug report from discovery to resolution.

  1. Classification: They categorized AI's role into three types: AI-discovered (AI finds the bug), AI-assisted (AI helps a human report a bug), and Suspected (unconfirmed AI use).
  2. Annotation: Given the scale (120k+ reports), they used a hybrid approach. They manually coded a pilot set to build a codebook, then used LLMs (GLM 5.3 and DeepSeek-V4.1) to annotate the full dataset, followed by human validation of the model outputs.
  3. Statistical Analysis: They used Firth logistic regression to identify factors associated with bug validity and cluster-bootstrap methods to compare AI-involved reports against traditional human reports within the same projects and timeframes.
Figure 2. The measurement framework used to analyze AI role, report reliability, and maintainer practices.

Datasets

The study analyzed 120,918 bug reports and over 500,000 follow-up comments from:

  • 101 Open-Source Projects: Including the Linux kernel and the top 100 most-starred repositories across eight languages (C, C++, Python, Java, JS, TS, Go, Rust).
  • Timeframe: January 2024 to September 2026.

Key Insights and Ideas

  • From Writing to Discovery: AI is no longer just a "secretary" for writing reports. While 83.6% of AI-involved reports are "assisted," the share of "AI-discovered" bugs (found via source code review) is growing rapidly, reaching nearly 24% of the AI-involved pool by late 2026.
  • The "Surface Presentation" Trap: AI-involved reports are significantly more "complete" than human ones—they are better structured and include more required elements. However, completeness does not equal validity. LLMs are excellent at making a report look professional even when the underlying technical claim is incorrect.
  • Task-Specific Success: Not all AI uses are equal. Using AI for reproduction and diagnosis is strongly associated with higher bug validity (Odds Ratios of 2.25 and 1.42, respectively). Conversely, using AI simply to write the report text provides no significant boost to validity.
  • Maintainer Bias: Maintainers spend more effort (more actions per report) on AI-assisted reports of human-discovered bugs. In contrast, fully AI-discovered bugs often struggle to get attention, likely because they lack the "real-world impact" context that a human reporter provides.
  • The Co-occurrence of Tasks: Reporters rarely use AI for just one thing. As shown in Figure 4, problem diagnosis is the "hub" task, frequently paired with fix recommendations and report writing.
Figure 4. Co-occurrence of AI tasks in assisted bug reports. Diagnosis: problem diagnosis; Fix: fix recommendation; Writing: report writing; Reproduction: reproduction and testing; Retrieval: information retrieval; Debugging: debugging support.

Key Results

  • Growth: AI involvement in bug reporting grew from 0.7% in 2024 to 8.3% in late 2026.
  • Reliability: Information completeness in AI reports improved by 5.41 percentage points per quarter. Validity among decided reports also rose by 4.89 points per quarter.
  • Rejection Rates: Maintainers reject 23.53% of AI-generated reproducers, compared to only 5.14% for non-AI reproducers. This suggests AI often makes ungrounded assumptions about execution context.
  • Maintainer Adoption: Maintainers are also adopting AI. AI-involved maintainer replies rose from 0.07% to 3.40% over the study period, primarily for code investigation and fix verification.
  • Practical Validation: By following their own "Reproducibility, Verification, and Accountability" guidelines, the authors submitted 53 AI-discovered bugs to real projects; 69.81% were accepted as valid.

Limitations and Future Work

The authors acknowledge that identifying "suspected" AI use is inherently difficult and may lead to some misclassification. Furthermore, the study focuses on public GitHub and Linux data, which may not represent private corporate environments.

Future work could explore the "arms race" between AI-generated bug reports and AI-powered triage tools, as well as the long-term impact of AI on the "human-in-the-loop" dynamics of open-source sustainability. The authors also suggest more research into "agentic" workflows where AI tools autonomously navigate codebases to verify their own findings before submission.

Summary updated

Abstract

AI-generated bug reports have drawn criticism for unreliable claims and faulty reproducers that burden maintainers. However, as LLM capabilities improve, some maintainers have begun to express more positive views on AI use in bug reporting and maintenance. Understanding this evolution requires examining whether AI-involved reports are becoming more reliable, which uses of AI are associated with valid findings, and how maintainers investigate these reports and act on them. To this end, we conduct the first large-scale study of 120,918 bug reports from 101 open-source projects between January 2024 and September 2026, tracing AI involvement from discovery and reporting through validation and resolution. Our findings show that AI-involved reports become more complete over time, with a growing share of reports confirmed as valid. In bug reporting, AI for reproduction and diagnosis are associated with higher validity, whereas report writing is not. Extends into maintenance, maintainers increasingly use AI to investigate code and recommend fixes. However, they devote more effort to AI-assisted reports of human-discovered bugs than to reports of AI-discovered bugs. These findings highlight both AI's growing participation in bug investigation and the continuing need for verification and follow-up. Drawing on these findings, we provide practical guidance for turning AI-generated findings into verifiable reports: emphasizing reproducibility, verification, and accountability. We followed this guidance when submitting 53 reports on AI-discovered bugs, of which 69.81% were accepted by maintainers as of writing.
BibTeX
@misc{liu2026copilot,
  author = {Yinuo Liu and Yunlong Lyu and Zichao Luo and Tian Dong and Zian Luo and Peng Chen and Hao Chen},
  title = {{Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and Maintenance}},
  year = {2026},
  eprint = {2611.25605},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25605},
}
Figure