KG-AST for Cheaper CI/CD: Budgeted Test Selection and AI Bug Review

Henry YeungarXiv 2026 · arXiv

TL;DR

KG-AST introduces a "budgeted" approach to CI/CD, using a machine learning model to rank the risk of code commits before running expensive tests or AI reviews. By focusing on just the riskiest 20% of commits, the system catches up to 64.6% of bug-introducing changes, significantly outperforming standard heuristics like "test the biggest changes first."

What Problem Does This Paper Address?

Modern software development relies on Continuous Integration (CI) and AI-assisted code reviews, both of which are computationally expensive and labor-intensive. Currently, most pipelines treat every commit the same: the full test suite runs, and the AI (or human) reviewer reads every diff.

However, the authors point out a glaring inefficiency: only a small minority of commits (4% to 35% in their data) actually introduce bugs. This means a massive portion of the CI budget is spent re-verifying "safe" changes. The challenge is to predict which commits are risky at commit time—before heavy tests run—so resources can be allocated where they are needed most.

Methods

The authors developed KG-AST, a system that operates in three main stages:

  1. Risk Scoring: Using a LightGBM gradient-boosted tree model, the system scores each commit based on 59 features extracted from Git history (e.g., lines changed, author experience, file age, and commit message cues).
  2. Budgeted Selection: Commits are ranked by risk. Only the top fraction (the "budget") receives heavy testing or AI review.
  3. Graph-Based Slicing: For selected commits, the system builds a "change radius" using a code knowledge graph. It identifies changed methods and their callers/callees. This slice is used to:
  4. Provide context to an AI reviewer (LLM).
  5. Perform static test selection (walking the call graph to find relevant tests).

Notably, the model is cross-language. A single model was trained on projects across 11 different languages, testing the hypothesis that "buggy-looking" code patterns transcend specific syntax.

Datasets

The study utilized a massive dataset of 155 open-source projects spanning 11 languages (including Java, Python, Rust, Go, and JavaScript).

  • Java Focus: 20 to 30 Java projects were used for deep-dive experiments.
  • Labeling: The authors used the SZZ algorithm, which identifies bug-introducing commits by tracing back from later "fix" commits using git blame.
  • Scale: Each project contributed its most recent 2,500 to 3,000 commits.

Key Insights and Ideas

  • One Model to Rule Them All: Surprisingly, a model that has never seen a specific language performs just as well as one trained on it. This suggests that the metadata of how we code (commit frequency, diffusion of changes, specific keywords) is a universal signal for risk.
  • Size Isn't Everything: While "biggest change first" is a common heuristic, KG-AST beats it by 15.1 points in Java projects. About two-thirds of the model's predictive power comes from change size, but the remaining third—derived from history and content—is what provides the competitive edge.
  • Graph Context Empowers AI: AI reviewers are significantly more effective when they see a "slice" of the call graph around a change rather than just the raw diff. Context increased the "exact find" rate of defects from 26% to 32%.
  • The "Negative" Results: In a refreshing display of scientific honesty, the authors report that several "sophisticated" ideas—like using Poincaré embeddings for risk scoring or complex hyperparameter tuning—actually added no measurable value. Simpler features like the number of if statements and the share of test code were far more predictive.

Key Results

The core metric used is coverage at a 20% budget: if you only have the resources to check 20% of your commits, what percentage of bugs will you catch?

  • Java Performance: In 20 Java projects, KG-AST covered 64.6% of bug-introducing commits at a 20% budget, compared to 49.5% for the "biggest change" baseline and 19.6% for random selection.
  • Language Consistency: The model beat the biggest-change baseline in all 11 languages tested.
Table showing performance across 11 languages. Note how the model consistently beats the "Biggest change first" baseline.
  • Feature Impact: As shown in the table below, adding "content of changes" and "message and time" features significantly boosts the Area Under the Curve (AUC) and coverage compared to using commit size alone.
Table breaking down how different feature groups contribute to the model's accuracy and coverage.
  • Test Selection: The static call-graph selector was able to include the specific test written for a fix in 85% of cases while running only ~8% of the total test suite.

Limitations and Future Work

The authors are transparent about several hurdles:

  1. Noisy Labels: The SZZ algorithm is known to be "noisy," sometimes blaming the wrong commits for a bug. While they guarded against this with stricter labeling tests, it remains a fundamental challenge in the field.
  2. Proxy Metrics: The study measures "share of commits" as a budget, not actual CI minutes or dollars saved. In environments where CI environments are "pre-built" and shared, skipping a commit might save less than expected.
  3. AI Judges: The audit of AI review findings relied on other AI models as judges, which may share the same biases as the reviewer.
  4. Future Directions: The authors aim to measure real-world cost savings by replaying actual CI logs and to extend the call-graph selection beyond simple unit tests to more complex API and flow tests.

Summary updated

Abstract

Continuous integration and AI-assisted review spend the same budget on every commit, although few commits introduce bugs. We study KG-AST, which scores each commit risk from git history and code-graph features before heavy tests or AI review run, so the budget goes to the riskiest commits first, and which hands the AI reviewer only a graph slice around the change. On 155 open-source projects in 11 languages, a single cross-language model trained on other projects ranks commits so that heavy testing of the riskiest 20% covers 49-64% of bug-introducing commits depending on the language, against 20% for random selection and 40-55% for testing the biggest change first. Every language beats the biggest-change baseline, by 3.8 to 13.2 points. In 20 Java projects the gain at a 20% budget is 15.1 points (64.6% vs 49.5%, 95% interval +11.6 to +18.4), and the model wins in 19 of the 20 projects. About twelve training projects suffice; richer features, tuning and cleaner labels did not help. An AI reviewer that sees only a commit's diff finds 26% of later-fixed defects exactly (32% with graph context), and about half of its findings that did not match a later fix were judged real by independent AI judges. A static call-graph selector runs about 8% of a test suite while including a test written for the fix in 85% of 67 fix commits. The Poincare embedding used in our retrieval work adds nothing measurable here: commit-risk AUC, bug review and severity ranking are unchanged or worse. The labels are noisy and the budget metric assumes a heavy test catches the bug it covers; no CI minutes were measured.
BibTeX
@misc{yeung2026kg,
  author = {Henry Yeung},
  title = {{KG-AST for Cheaper CI/CD: Budgeted Test Selection and AI Bug Review}},
  year = {2026},
  eprint = {2611.06950},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06950},
}
Figure