Schedule Luck in Concurrent Coding Agents

When several AI coding agents work on the same repository at once, the result can depend on timing. One agent might read a file before or after another agent edits it, or a test might run before or after a patch lands. Current benchmarks report one pass rate per system. That number mixes three sources of variation: the language model's sampling, drift in the environment, and the order in which concurrent events happened to occur. Containers fix the starting state of a run, but they do nothing about ordering. This proposal builds the concurrent agent system on the actor model, an old idea from distributed computing in which every component is an isolated process with a private inbox that changes only in response to messages it receives. It then logs exactly which message each inbox received, in what order, and what outside input came in. With that record, any run can be replayed exactly. More usefully, the record shows which events were truly concurrent, so the system can deliberately re-run a task under the other orderings that could have happened. The goal is to measure how much of a multi-agent system's benchmark score comes from lucky interleavings, and to give developers a way to reproduce and fix the unlucky ones.

In two sentences: treat each agent and each tool as an actor that owns its own piece of the sandbox, and record only inbox arrival order plus outside inputs such as model outputs, clocks, and network responses. That record makes concurrent agent runs replayable and lets us search over alternative orderings, which gives the first direct estimate of "schedule flakiness" in agent benchmarks.

Core Claim

The actor formalism of Hewitt, Bishop and Steiger and the analysis in Modelling Distributed Systems establish a useful property. If actors share no memory, each actor's state is fully determined by the sequence of messages it has received. Everything else is deterministic computation. This paper supplies our method: it tells us exactly what must be logged.

Agent sandboxes today break this property. OpenHands runs a sequential event stream against a single shared Docker container. When agents are delegated to concurrently, they touch the same files through a channel the log never sees. We make the environment itself part of the actor graph. A repository actor owns the working tree and applies edits only in response to messages. Each tool actor (shell, test runner, interpreter) works in a private copy-on-write layer and publishes results as messages. Once the filesystem is behind inboxes, every race over shared state shows up as two messages arriving at the same inbox in some order.

The claim has three parts, each testable. First, logging per-inbox arrival order together with outside inputs is enough for exact replay of concurrent agent runs, at an overhead small compared with model inference. Second, the partial order recovered from these logs identifies which reorderings could actually have happened. This lets us explore alternative schedules far more cheaply than blind reruns. Third, on concurrent versions of established coding agents, a measurable share of benchmark instances are schedule-flaky: they pass under some feasible orderings and fail under others, even with model sampling fixed.

If the third part fails, the contribution shrinks to a replay tool, and we will say so.

Research Program

We build a small actor runtime that runs inside a pinned container image. Each agent, tool, and the repository is an actor with an inbox. The runtime writes an append-only log in the spirit of the JSONL provenance log in SciAgent's audit harness, with two differences. SciAgent logs top-down parent–child sessions. Our log instead records, for every inbox, the identity and order of delivered messages, and for every boundary with the outside world, the raw input: model completions, timestamps, random seeds, and network or subprocess outputs.

Replay re-injects the outside inputs and forces the recorded delivery order at every inbox. Nothing else needs to be stored. From the logged "who-sent-what-after-seeing-what" relation, we compute which deliveries at the same inbox were causally unordered, meaning they genuinely raced.

Schedule exploration then works as follows. We replay a run exactly up to a chosen race. We swap the delivery order of the two racing messages. We then continue live, with model calls at temperature zero and fixed seeds, so that divergence after the swap is attributable to the reordering. We compare this with a sampling perturbation: keep the recorded order but change the model seed at the same point.

Comparing the two gives a per-instance decomposition of outcome variance into three parts: ordering, sampling, and a residual we attribute to the environment. The residual is measured by replaying identical logs on fresh containers. Because the log ties every change in the final repository to the message that caused it, the same record also provides trajectory provenance. This speaks to the disagreement between Agent-Diff, which argues that only before-and-after state differences should be graded, and SciAgent, which shows that matching outcomes can hide fabricated or degraded execution paths. We keep state-difference grading for pass or fail and use the message log only for diagnosis, so valid alternative paths are not penalized.

The systems under test are concurrent actor versions of two published designs. The first is the five-role pipeline of CodeR, in which the Reproducer and Fault Localizer run concurrently rather than in CodeR's static synchronous task graph. The second is the localization debate of SWE-Debate, with debaters proposing asynchronously instead of in orchestrated rounds. We also run a plain parallel-subagent scaffold on Terminal-Bench tasks.

Before claiming that any schedule effect comes from concurrency, we confirm that each actor port reproduces its original system's pass rate within noise when run with a single serialized order.

Baselines

The naive reference point is rerunning live. We launch the same system on the same pinned container image with the same seeds, record nothing beyond ordinary logs, and measure how often two runs produce the same message sequence and the same final repository. This is what benchmarks effectively do today. Habchi et al. report that practitioners treat rerunning as a band-aid that hides non-deterministic defects. We expect low exact-reproduction rates here. The replay system beats this baseline only if it reproduces essentially all runs exactly.

The strongest existing mechanism for replay is a single globally ordered event log, as in OpenHands's event stream, extended to record every model completion and tool output. We implement it on the same runtime, same container images, same models, and same instances, so the only difference is total order versus per-inbox order. We expect it to match our system on exact replay, and we concede that point in advance. Our system beats it only on exploration. At an equal budget of re-executions, per-inbox logging must find more distinct outcomes, and in particular more flipped pass/fail results, than reorderings drawn blindly from the total-order log. Many of the total-order log's reorderings are infeasible (they violate causality) or change nothing (they swap independent events).

The published pass rates anchor the task-performance side. With DeepSeek-V3, SWE-Debate reports 41.4% Pass@1 on SWE-bench-Verified, against 38.8% for OpenHands and 35.4% for SWE-Search in the same paper. CodeR reports 28.33% on SWE-bench Lite, and its own ablation drops from 22% to 10% with GPT-4o when the multi-agent structure and task graphs are removed. The strongest reported Terminal-Bench result is 62.9% for GPT-5.2 with the Codex CLI scaffold. We do not aim to beat these numbers. We aim to show that for concurrent systems, a single reported number hides a schedule-dependent spread.

Two ablations isolate the key ingredient. In the first, we keep inbox logging but let tool actors share one container filesystem, as OpenHands does. This tests whether routing environment state through actors is what makes filesystem races visible and replayable. In the second, we log outside inputs but not delivery order. This tests whether ordering information is needed at all.

Why not just…?

Why not just cache model responses and pin the container? For a sequential agent, this probably suffices, and we do not claim otherwise. With concurrency, the order in which agents see each other's results changes their prompts. A prompt-keyed cache then misses, and the run diverges live. The "outside inputs only" ablation measures exactly this. If it reaches the same replay fidelity as full logging on our concurrent systems, the ordering log is unnecessary, and we will report that.

Why not just serialize the agents in a fixed round-robin order? This is deterministic by construction and is the most serious threat to the proposal. If serialized execution gives the same pass rate at an acceptable wall-clock cost, the practical advice is simply "don't run agents concurrently." We run the serialized variant on the same instances and report pass rate and wall-clock time side by side. The schedule-exploration results also answer a related question: whether the one fixed order is good, bad, or typical within the distribution of feasible orders. That answer is informative even if serialization wins.

Why not just rerun many times and report mean and variance? Repeated live runs tangle ordering, sampling, and environment drift, so they cannot say why an instance is flaky or reproduce a specific failure for debugging. Zhu et al. find that 70% of audited agent benchmarks violate task or outcome validity, often through environment leakage and drift. Variance alone cannot separate that from model behavior. We include matched-budget blind reruns as a comparison for flakiness detection.

Why not just grade state differences and ignore trajectories? We do grade by state difference. The message log is used only to explain outcomes, not to score them. SciAgent shows why the explanation matters: runs with identical outcomes (a 1.000 mapping rate for both systems on BRCA1) hid a silently degraded execution path, and its verifier cost only 1.7% to 9.9% of total compute.

Proposed Experiments

Because repeated schedule exploration on long tasks is expensive, and Terminal-Bench attempts can run up to two hours, we use a fixed random subset of 100 SWE-bench-Verified instances and all 89 Terminal-Bench tasks. Each instance gets one recorded run, one exact replay on a fresh container, and a fixed budget of reordering and reseeding branches chosen by each exploration method.

We measure five things:

  1. Exact-replay fidelity: whether the replay produces an identical inbox log and final state hash.
  2. Logging overhead: bytes stored per run and added wall-clock time relative to model inference time.
  3. Schedule-flakiness rate: the fraction of instances whose pass/fail flips under some feasible reordering at fixed sampling.
  4. The variance decomposition into ordering, sampling, and environment.
  5. The number of distinct final outcomes found per re-execution, compared between per-inbox and total-order exploration.

The decisive checks are these. Replay fidelity must be near-total for full logging and clearly lower for live reruns and response caching. Per-inbox exploration must find flips more efficiently than total-order or blind exploration at equal budget. The shared-filesystem ablation must lose fidelity, which would show that putting the environment behind actors is what does the work. Finally, the schedule-flakiness rate must be large enough to move a reported pass rate by more than the gaps separating published systems, such as the 2.6 points between SWE-Debate and OpenHands. If flakiness is negligible and serialized execution costs little, the honest conclusion is that concurrency in current coding agents is unnecessary, and the replay runtime stands only as a debugging tool.