Do Coding Agents Understand Why Systems Crash---and Why They Don't?

Chenxi Huang, Haonan Wang, Jinghe Zhang, Aleksandr Nogikh, Franjo Ivančić, Petros Maniatis, Eugene Wu, Kostis Kaffes, Junfeng Yang, Baishakhi RayarXiv 2026 · arXiv

TL;DR

Researchers have developed a new way to test if AI coding agents actually understand program execution or are just "guessing" based on suspicious-looking code. By using source interventions—pairing a crashing version of code with a nearly identical fixed version—they found that even top-tier agents like Claude Opus 5 frequently miss real crashes and struggle to reason about bugs that manifest far from their root cause.

What Problem Does This Paper Address?

We are seeing a surge in AI "coding agents" capable of fixing bugs in real-world repositories. However, a successful patch doesn't necessarily mean the AI understands why the code failed. It might just be recognizing common "code smells" or patterns it saw during training.

This is particularly dangerous in complex systems like the Linux kernel. If an agent "fixes" a crash without understanding the underlying execution logic, it might inadvertently break intended functionality or introduce subtle regressions. Existing benchmarks often focus on small code snippets or require expensive step-by-step execution tracing, which doesn't scale to massive, multi-subsystem environments like an operating system kernel. The authors ask: Can these agents perform "execution reasoning"—predicting if a specific input will cause a crash based solely on the source code?

Methods

The authors introduce a paired evaluation methodology called source interventions. Instead of asking an agent to look at one piece of code, they present it with two versions:

  1. $T_c$ (Crashing Tree): The original code that crashes on a specific input.
  2. $T_n$ (No-Crash Tree): A version differing by exactly one commit (a fix) that does not crash on that same input.
Figure 1: Differencing over a pair with opposite crash outcomes isolates true discriminability. Tc and Tn differ only by ∆at the intervention site. When sampling coding agent to predict “crash or not” multiple times, trajectories blind to the site predict a crash equally often in both trees and cancel in Tc −Tn.

The agent must predict if a crash occurs, where it happens (the stack trace), and why. By comparing the agent's predictions on both versions, the researchers can calculate discriminability ($J$). If an agent predicts a crash on both the buggy and the fixed code, it isn't reasoning; it's just flagging "suspicious" code. A high $J$ score indicates the agent truly understands how the specific code change affects the execution outcome.

To dig deeper, they used an LLM-based "annotator" to scan the agents' internal reasoning logs (Chain of Thought) to see if they actually looked at the intervention site (where the fix was) and the crash site (where the failure manifested).

Datasets

The study introduces SYSRB (System Reasoning Benchmark), which contains 934 execution-validated source-tree pairs.

  • Linux Split: 545 pairs derived from syzbot (Google's continuous fuzzer for the Linux kernel).
  • OSS-Fuzz Split: 389 pairs from various open-source system libraries.

Unlike previous benchmarks, every pair in SYSRB is verified by actually running the code in a Docker container to ensure the "fix" truly stops the crash and the "bug" truly triggers it.

Key Insights and Ideas

  • The "Suspicious Code" Bias: Agents often suffer from a "false alarm" problem when they see code that looks messy. If an agent analyzes a crash site but misses the fix (the intervention site), it often predicts a crash on both versions. This proves that single-tree evaluations (only testing on buggy code) vastly overestimate an AI's reasoning skills.
  • Observation $\neq$ Understanding: Simply finding the relevant code isn't enough. Even when agents like Claude Opus 5 observed the intervention site, their ability to correctly discriminate between the crash and the fix was only about 75%. For other models, this "conditional discriminability" was much lower.
  • The "Distance" Penalty: Reasoning becomes significantly harder when the root cause and the crash manifestation are far apart. If a bug in one Linux subsystem causes a crash in a completely different one, the agents' performance drops sharply.
Figure 4: JO∆=T versus Pr(O∆= T)

Key Results

The researchers tested four agents: Claude Opus 5, GLM 5.3, DeepSeek V4 Flash, and Gemini 3.8 Flash.

  • Claude Opus 5 was the top performer, achieving a total discriminability ($J$) of 67.7%. However, it still missed over a quarter of all real crashes.
  • Conservative Bias: All agents were heavily biased toward predicting "no crash." They rarely raised false alarms (low False Positive Rate), but they frequently failed to spot actual vulnerabilities (high False Negative Rate).
  • Localization: When agents did correctly predict a crash, they were generally good at identifying the correct crash site. As shown in Figure 3, Claude Opus 5 matched the exact crash frame in about 50% of its successful predictions.
Figure 3: Frequency (%) of stack trace score on Tc

Limitations and Future Work

  • Data Contamination: Since the Linux kernel and OSS-Fuzz commits are public, there is a risk that the models saw these specific fixes during their pre-training phase.
  • Environment Restrictions: In these tests, agents were restricted to basic file navigation and bash commands. In the real world, developers use debuggers and version control history, which might change how agents perform.
  • Cost: Running these evaluations is expensive. Each "task" involves multiple trajectories of long-context reasoning, which can be constrained by API budgets.

The authors suggest that future work should focus on giving agents better tools (like actual execution feedback) and developing training methods that specifically target "execution reasoning" rather than just pattern matching.

Summary updated

Abstract

Coding agents are increasingly capable of fixing real-world bugs, yet they still struggle with large systems codebases such as the Linux kernel. To understand why, we test how agents reason about execution in these codebases through crash reasoning: predicting from source code whether a given input crashes the system. A correct prediction alone, however, does not establish this ability, since it may come from merely spotting suspicious code. In this paper, we propose source interventions: we pair source trees that differ by a single commit, so that the same input crashes only one of them, and test whether agents predict crashes more often on the crashing tree. Experiments with four coding agents on SysRB, our dataset of 934 such pairs from Linux and OSS-Fuzz projects, yield three main findings. First, agents are biased toward predicting no crash: they rarely raise false alarms but miss many real crashes, and even the best agent misses over a quarter of them. Second, locating the code that determines the crash outcome is not enough: agents often misjudge its effect, especially when the crash occurs far from that code. Third, testing only on crashing code is misleading: agents that merely notice suspicious code predict crashes on both source trees, which would look like correct reasoning on the crashing tree alone. Together, these findings indicate that current coding agents cannot yet reliably reason about execution in complex systems codebases, and that paired evaluation is necessary to measure this ability.
BibTeX
@misc{huang2026do,
  author = {Chenxi Huang and Haonan Wang and Jinghe Zhang and Aleksandr Nogikh and Franjo Ivančić and Petros Maniatis and Eugene Wu and Kostis Kaffes and Junfeng Yang and Baishakhi Ray},
  title = {{Do Coding Agents Understand Why Systems Crash---and Why They Don't?}},
  year = {2026},
  eprint = {2610.29332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29332},
}
Figure