Execution-Guided Watermark Backoff for Code Repair Loops

A watermark for AI-generated code is a hidden statistical signature. While the model writes, it quietly favors a pseudorandom "green" subset of tokens, and a detector that knows the secret key can later count green tokens to decide whether the code came from the model. The trouble with code is that most tokens are forced by syntax. Pushing the model toward green tokens at those positions breaks programs, so every existing code watermark caps its strength to protect first-try correctness. Coding agents now rarely stop at a first try. They run their code in an isolated container, read the error, and patch it. This proposal asks whether that execute-and-repair loop changes the watermark trade-off. If the sandbox catches the watermark's mistakes, the watermark can afford to be much stronger. When a test fails, the agent can weaken the watermark only in the lines the failure points to and leave it at full strength everywhere else.

The claim, in two sentences: in an execute-and-repair loop, draft the code with an aggressive watermark, then use execution feedback (tracebacks and which lines the failing tests cover) to back the watermark off locally in the regions implicated in failures. At a matched number of sandbox executions, this should produce more detectable final programs than resampling fully watermarked drafts until one passes, with the same correctness on held-out tests.

Core Claim

Current entropy-constrained code watermarks decide where to embed signal only from the model's next-token uncertainty or from static code structure. SWEET skips positions whose next-token entropy falls below a threshold. ACW uses a trained network over syntax-tree-equivalent variants. Both are tuned for one-shot generation, where any correctness loss is final. The methods that do use execution, CodeTracer and SWaRL, use unit tests only as an offline reinforcement-learning reward and then generate in a single pass at inference. Meanwhile, repair systems such as OpenCodeInterpreter and OpenHands run multi-turn sandboxed debugging with no watermark at all.

We claim that runtime feedback at inference time is a better signal for where a watermark may safely go than entropy alone. The reason is that it measures the thing the watermark actually risks, which is a change in behavior. A secondary claim concerns detection. Repair turns are generated from a context full of error messages that the detector will never see, so a watermark that works in a loop must compute its gating decisions from something the detector can reproduce. We treat that as a first-class design constraint.

Research Program

The base scheme is green-list biasing in the style of Kirchenbauer et al., with SWEET's entropy gate. Detection uses the standard one-tailed test over the $T$ scored positions:

$$z = \frac{|s|_G - \gamma T}{\sqrt{T\gamma(1-\gamma)}}$$

Here $|s|_G$ is the number of green tokens among scored positions and $\gamma$ is the green-list fraction. Signal accumulates only where entropy allows it, as Thickstun et al. formalize, so low-entropy code gives a small $T$ and a weak $z$. Our lever is to raise the per-position bias $\delta$ and lower the entropy threshold $\tau$ well beyond what one-shot methods tolerate. The loop then pays the resulting correctness cost.

The loop has three parts.

Drafting. The first draft is generated at an aggressive setting $(\delta_{hi}, \tau_{lo})$.

Localization. When the draft fails in the sandbox, we locate the fault using the traceback line together with line coverage of passing versus failing visible tests. Each line gets a suspiciousness score, higher for lines executed by failing tests and not by passing ones. This matters because dynamic coverage is not reliably predictable from static reading, as Predicting Code Coverage without Execution indicates.

Backoff repair. The repair turn is constrained to a minimal diff over the implicated lines. Inside those lines, the watermark drops to a conservative setting. Outside them, tokens are copied verbatim, so their green status is preserved. Because the green-list seed hashes the preceding file token, verbatim copying keeps every seed intact except at the edit boundaries.

Detector consistency is the subtle part. SWEET's detector recomputes entropy from the task prompt and the code prefix. During a repair turn, however, the generating model sees a conversation containing stack traces. We therefore compute the gating entropy from a prompt-plus-file-prefix context that the detector can reconstruct. The bias is still applied to the full-context logits. This costs one extra forward pass per token, which we report.

We also answer the purely measurement question the field lacks: how much of a SWEET, ACW, or CodeTracer watermark survives unmodified multi-turn repair. The answer calibrates how much of our gain comes from locality versus strength.

Baselines

The naive reference is SWEET applied inside an OpenCodeInterpreter repair loop, at the one-shot-tuned threshold, with free-form repair and no backoff. This is what a practitioner would do by default. The strongest reported one-shot numbers anchor it. SWEET reports AUROC 0.943 on HumanEval with Pass@1 within 2–4% of unwatermarked (Who Wrote this Code?). OpenCodeInterpreter-33B reports HumanEval Pass@1 rising from 79.3 single-turn to 82.9 with execution feedback (OpenCodeInterpreter). We found no reported result on Lacuna for any watermark evaluated after multi-turn repair.

The strongest competitor is best-of-$k$ watermarked resampling. We sample SWEET-watermarked programs at a fixed global strength, execute each in the sandbox, and return the first that passes the visible tests. It is dangerous because every returned program is fully watermarked and its correctness equals pass@$k$. We run it with the same generator, the same visible tests, and the same cap of $k$ sandbox executions as our loop. We tune its $(\delta, \tau)$ with a grid search of the same size we give our method, on a held-out split of MBPP. No reported result was found on Lacuna for watermarked best-of-$k$ on these benchmarks.

We also compare against CodeTracer used as a one-shot generator. It reports 62.65% Pass@1 and 77.71% AUROC on HumanEval (CodeTracer). We additionally run it as the drafting policy inside our loop.

Two ablations isolate the key ingredient:

  • Random-region backoff weakens the watermark on the same number of lines, but chooses them at random rather than from execution feedback.
  • Global backoff weakens the whole program after any failure.

We beat the field if we reach higher true-positive rate at 1% false-positive rate than best-of-$k$ at the same sandbox budget, with final pass rate on the stricter HumanEval+/MBPP+ tests within one point. We must also beat random-region backoff by a clear margin; otherwise locality is not what helps.

Why not just…?

Why not just resample fully watermarked drafts until one passes? This is the real threat. On short HumanEval functions it may simply win, because resampling never dilutes the signal. Our bet is that best-of-$k$ must choose one global strength weak enough that some sample passes. Localized backoff pays the correctness cost only where the watermark actually changed behavior. The matched-budget comparison settles this directly. If best-of-$k$ matches us, we will report that sandbox calls are better spent on resampling for function-level tasks. We would then move the method's claim to multi-checkpoint settings, where resampling an evolving codebase is not possible.

Why not just repair normally and watermark the final passing code afterwards by rephrasing it? Post-hoc watermarking by rephrasing tried exactly this on HumanEval and MBPP. It reports a true-positive rate of about 0.3 at low false-positive rate on Llama-3.1-8B, along with Pass@1 drops, because rephrasing code has no execution check. CodeTracer reports post-hoc code detectors near 50% AUROC. We include an execution-verified variant as a baseline: rephrase, keep only passing rewrites, cap at the same budget. This tests whether the sandbox rescues post-hoc embedding.

Why not just crank up the watermark and let the unmodified repair loop fix the breakage? Unconstrained biasing is known to be costly. WLLM's Pass@1 fell to 30.82% in SWaRL. Free-form repair can also rewrite far more than the broken line, destroying signal elsewhere. The global-backoff ablation and the survival study measure exactly how much signal free-form repair erases.

Why not just hide the watermark in comments? Suresh et al. and MCGMark both note that comments and strings are trivially stripped. We include comment-only watermarking in the robustness table to show this, rather than assume it.

Proposed Experiments

We use OpenCodeInterpreter models, which were trained for execution-feedback repair, at 6.7B and 33B, with the Code-Feedback conventions. We evaluate on HumanEval and MBPP, with the visible tests used inside the loop. Correctness is scored on HumanEval+ and MBPP+ hidden tests, which exposes any semantic drift that backoff hides from the visible tests. Metrics are:

  • AUROC and true-positive rate at 1% false-positive rate;
  • final pass rate within $T \le 5$ turns;
  • sandbox executions and generated tokens.

False positives are calibrated on human canonical solutions and on unwatermarked outputs from the same repair loop.

The first decisive check is the matched-budget comparison against best-of-$k$ at $k \in \{1,3,5\}$. The second is the random-region ablation. The third applies the Rename, AddDeadCode, InsertPrint, and WrapTryCatch transformations from Suresh et al. to final programs. Under AddDeadCode, their Unigram true-positive rate fell from 0.76 to 0.01. We predict that the higher per-position strength our loop permits raises post-attack detection.

The fourth moves to SlopCodeBench, where agents extend their own code across 196 checkpoints and even the best agents pass only about 14.8%. Here we measure watermark survival across checkpoints and whether watermarking worsens the structural erosion and verbosity growth the benchmark already documents for unwatermarked agents. A watermark that survives but makes agent code measurably more eroded would count against the method, and we will report it as such.