Learning When to Commit a Security Patch

Security fixes reach users late. Over half of security patches take more than 180 days to be ported from Vim to its fork Neovim (PPatHF), and the gap between discovering a bug and deploying a patch averages 30 days or more (VulShield). Language-model agents that write or port patches could shorten this window. They share a failure that is well documented in navigation and desktop automation: they do not know when they are done. Some keep searching until they run out of budget. Others stop early and declare success on a patch that only looks right. In security the second failure is the worse one. A patch that is falsely marked "fixed" closes the ticket and quietly extends the exposure, while an agent that says "I could not fix this" at least sends the bug back to a human.

This note proposes to train a repair and patch-porting agent with reinforcement learning (RL), meaning the model improves by trying tasks and being scored on outcomes, so that it decides by itself when to commit a patch. Three things make this different from existing early-exit work. The penalty is asymmetric: a false "fixed" costs more than giving up. The agent may abstain explicitly. And credit for the stopping decision is assigned in hindsight, by checking after the fact which intermediate patch states would already have passed cheap verification. A layered verification scheme, cheap structural checks on every rollout and expensive runtime checks only on committed candidates, is meant to make this affordable, because the cost of RL with execution feedback is the reason prior repair RL settled for noisy LLM judges.

Core Claim

Our hypothesis is that the stop decision in agentic vulnerability repair is a separately learnable skill. It can be trained without injecting "stop now" prompts. Training it should cut false completions and wasted exploration together, and it should reduce RL training cost, not add to it. Concretely, we expect an agent trained with asymmetric commit rewards and hindsight stop credit to show a lower false-commit rate and fewer tool calls than two alternatives: an agent trained with a plain per-step length penalty, and an agent imitating trajectories truncated at their first verified-good state. We expect this at equal or better verified repair rate.

Where This Sits

Evidence that the problem exists. Three results motivate the proposal.

  • Stopping is a bottleneck in its own right. Giving a navigation agent an ideal external stop signal raises completion by over 200% (Xiang et al.).
  • On desktop tasks, unverified agents that fail almost always believe they succeeded (GUI-Pro-Agent).
  • In incident remediation, non-reasoning agents fail mostly by exhausting their iteration budget, in 9 of 15 runs (Compositional Investigation). That study uses five incidents and trains nothing.

The theory we adopt treats termination as a decision made from the agent's internal belief. The agent should stop when the expected value of committing exceeds the expected value of continuing (Hansen; Rao).

The two closest methods. Both address early exit in reasoning models.

  • DEER exits at test time when the confidence of a trial answer crosses a threshold. It trains nothing, it is sensitive to calibration ("shy" models never exit), and its authors name RL-time integration as future work.
  • S-GRPO does integrate exit into RL. It works by injecting an external "Time is limited" prompt at intermediate points and rewarding the earliest correct answer with decaying reward. Its tasks are single-turn math problems with scalar answers, and the prompt, not the policy, chooses where exits are tried.

Our proposal differs from both. It works in multi-turn tool environments. The commit action is chosen by the policy itself. And the reward separates wrong commits from abstentions, which neither method models.

Methods we reuse. - From VulAgentRL we take code-property-graph tools and deterministic graph-node checks as a cheap verifier. A code property graph is a queryable graph of a program's calls and data flow. VulAgentRL showed that GRPO (a variant of RL that scores each sampled attempt against the average of a group of attempts on the same task) reduced tool queries from 7.6 to 4.3 in vulnerability detection. That is an incidental termination effect in detection, not an objective aimed at stopping in repair. - From VulDebugger and VulShield we take sanitizer-derived crash-free constraints. These are checks that the exploit input no longer triggers a memory error, and we use them as the expensive, high-fidelity verifier.

Constraints on the design. - Vul-R2 found that RL on repair makes responses grow longer and training slower. It relied on an LLM critic plus character matching. - PPOCoder and CoTran showed that running compilers and solvers inside every rollout stalls training. - VulAgentRL, Vul-R2 and CoTran all report that RL from a cold start fails without a supervised warm start, so we include one.

The multimodal reasoning gap as a lens. We use the work on the multimodal reasoning gap for its methodology, not its modality. GeoSQA separated perception from reasoning by handing models human transcriptions of diagrams, and models still failed. MATH-Vision likewise located failure after components were recognized. We apply the same controlled substitution to termination. We give the agent oracle evidence (the true root-cause location) and measure whether it still fails to commit correctly. That separates "could not find the evidence" from "found it but could not decide." We deliberately do not render code graphs as images. The GeoSQA-style result suggests that better perception would not fix a reasoning and decision failure, and a visual channel would add cost to an already expensive loop.

Research Program

The agent. It is an open 7B-scale code model. It has repository tools (read, search, graph queries), a build tool, and three terminal actions:

  • COMMIT(patch), which submits a patch;
  • ABSTAIN(reason), which escalates the bug to a human;
  • continuing to act, up to a hard budget. Running out of budget counts as abstention, with an extra small penalty.

The reward. Each episode ends in one of four outcomes: a verified commit, a false commit (it fails full verification), an explicit abstention, or budget exhaustion. The episode reward is

$$R = \mathbb{1}[\text{verified commit}] - \alpha\,\mathbb{1}[\text{false commit}] - \beta\,\mathbb{1}[\text{abstain}] - \gamma\,\mathbb{1}[\text{budget exhausted}],$$

with $\alpha > 1 > \gamma > \beta \ge 0$.

  • $\alpha$ is the penalty for a false commit.
  • $\beta$ is the small cost of an explicit abstention.
  • $\gamma$ is the cost of running out of budget, set above $\beta$ so that a clean abstention beats wandering.

Under this reward the policy should commit only when its estimated probability $p$ that the patch verifies satisfies $p \cdot 1 - (1-p)\alpha > -\beta$. Rearranged, the threshold is $p^* = (\alpha - \beta)/(1+\alpha)$. This is the belief-boundary rule from Hansen, with the threshold set by the security cost of false fixes rather than tuned by hand.

Hindsight stop credit. This is the key ingredient. After each rollout we replay the sequence of patch states the agent held and run only the cheap verification tier on each one: compile, the structural graph checks, and ported regression tests where they exist. These checks run in CPU workers in parallel with GPU training. Let $t^*$ be the first step at which the patch in hand passes the cheap tier and would later pass full verification. Two cases get an extra advantage signal on the COMMIT action at $t^*$:

  • if the agent continued past $t^*$, the signal pushes toward committing (the last-mile failure);
  • if the agent committed before any state passed, the signal pushes against committing (the hallucinated-success failure).

No text is inserted into the rollout. The policy's own trajectory is only re-scored, which is the main contrast with S-GRPO.

Layered verification. Full dynamic verification means building the program, running the proof-of-concept exploit under AddressSanitizer, and running the project tests. It is applied only to committed patches that pass the cheap tier. We will log how often each tier runs and how much wall-clock time it takes. That lets us report whether the combination of shorter trajectories and gated verification lowers GPU-hours per unit of improvement, compared with verifying every rollout the way PPOCoder does.

Training. Training starts with supervised warm-start trajectories distilled from a strong commercial model, as VulAgentRL does, followed by GRPO. Training environments come from two sources:

  • executable Juliet cases;
  • Vim-to-Neovim porting commits dated before a cutoff, verified by build and tests.

Proposed Experiments

Primary evaluation. The main in-distribution test is held-out Vim-to-Neovim patches after the cutoff, drawn from the 310-patch PPatHF set. The out-of-distribution test is the 50 real-life CVEs from VulDebugger, spread across libtiff, libxml2, gpac and jasper, which come with exploit inputs and so support full runtime verification. The rca-llms incidents serve as a small transfer probe, to check whether the learned stopping behavior carries over to remediation that is not memory-safety repair.

Metrics. We report:

  • verified repair rate;
  • false-commit rate, meaning the share of commits that fail full verification;
  • abstention rate;
  • last-mile rate, meaning the share of trajectories that passed through a verifiable state but did not commit;
  • tool calls and tokens per task;
  • verifier-seconds and GPU-hours per training step.

On the porting set we also report exact syntactic match, so results are comparable to PPatHF. As a security-facing summary we report an expected-exposure proxy: tasks falsely closed count as unresolved for the whole remaining window, while abstained tasks go back to the human queue.

Oracle-evidence study. We rerun each trained agent with the true fault location supplied. If false commits persist at similar rates, the bottleneck is the decision itself, and hindsight stop credit should be the ingredient that helps.

Baselines

The naive reference point. This is the warm-started supervised agent run with a fixed step budget. It commits whenever it chooses and has no abstain action. It fixes the floor for both false-commit and last-mile rates. Any trained variant has to beat it on verified repair rate without raising false commits.

The strongest external systems. There is one per benchmark.

  • On Vim-to-Neovim, PPatHF reports 42.3% exact syntactic match, against 4.2% for FixMorph.
  • On the 50 real-life CVEs, VulDebugger reports a 60.00% (30/50) repair success rate, against 28% for AutoCodeRover. VulDebugger uses commercial models in long interactive debugger sessions with no stopping mechanism, which makes it the accuracy ceiling to approach and the cost profile to undercut.

We will rerun PPatHF's released pipeline on our post-cutoff split rather than rely only on its reported number. For VulDebugger we will quote the reported figure and also run a same-base-model reimplementation of its debugger loop. We count "beating" VulDebugger as matching its repair rate within a few tasks at a small fraction of its tool calls, and with a false-commit rate we can actually measure, which it does not report.

The strongest early-exit competitor. This is S-GRPO adapted to agents: at sampled intermediate turns, insert its "Time is limited" instruction, force a commit, and apply its decaying reward $1/2^{N-1}$. It is trained from the same warm start on the same environments, with the same number of rollouts and GPU budget, and with hyperparameters tuned over the same grid size as ours. S-GRPO reports cutting tokens by 35.4% to 61.1% and gaining up to 6.08 points of accuracy on DeepSeek-R1-Distill-Qwen-7B in math. No reported result exists on Lacuna for any early-exit method on vulnerability repair. We will also run DEER at test time on our supervised agent, sweeping its confidence threshold.

Ablations of the key ingredients. Three ablations isolate them:

  • asymmetric reward with no hindsight credit;
  • hindsight credit with symmetric reward ($\alpha = 1$, no abstain action);
  • our full method with every rollout sent to full verification, which isolates what the layered verification contributes to training cost.

Why not just…?

Why not just add a per-step cost to GRPO? This is the simplest threat. A length penalty will shorten trajectories, and it may reproduce VulAgentRL's drop from 7.6 to 4.3 tool calls. A uniform step cost does not distinguish stopping too late from committing a wrong patch. If anything, it rewards committing early, which raises false commits. We run it as a baseline, sweeping its coefficient. If at matched tool-call counts it reaches our false-commit rate, the asymmetric reward and hindsight credit are unnecessary, and we would report that.

Why not just gate commits with the verifier at test time? Where an exploit input exists, one can loop until the patch passes, as VulDebugger effectively does, or add a completion judge as GUI-Pro-Agent does. This is a real threat to the false-commit claim on the 50-CVE set. Our answer has three parts.

  • Test-time gating does nothing about wandering.
  • Its cost scales with the number of verifier calls. GUI-Pro-Agent's verifier reduced false completion by only up to 3.9% while adding judge latency.
  • Many fork ports lack an exploit input, so full verification is unavailable at deployment.

We compare verified repair rate, false commits and verifier calls between our agent without gating and the supervised agent with gating. We also report our agent with gating, to test whether the learned stop policy mainly saves cost or also improves accuracy.

Why not just imitate trajectories cut at their first verified-good state? Hindsight truncation followed by supervised fine-tuning is cheap, fully parallel, and plausibly teaches stopping. It is our strongest simple alternative, and we train it on exactly the trajectories our hindsight labeler produces. The evidence that imitation alone falls short in this domain is indirect. VulAgentRL's RL agent roughly doubled the supervised F1 under class imbalance (0.194), and CoTran's RL added 14.89 points of functional equivalence over supervised training. Neither result is about stopping. If truncation-based imitation matches our false-commit and last-mile rates, the contribution shrinks to the labeling procedure, and we would say so.

Why not just threshold confidence at test time? DEER's own analysis shows that thresholds fail on poorly calibrated models. The token-probability confidence of a whole patch is also not obviously meaningful. The threshold sweep will show whether any single setting reaches our operating point, measured as false commits against repair rate.

Decisive Checks

The proposal succeeds if two conditions hold together on the held-out ports and the 50 CVEs.

  1. Against both the per-step-penalty baseline and the truncation-imitation baseline, at matched tool calls, the full method lowers false-commit rate and last-mile rate at no loss in verified repair rate.
  2. The layered-verification ablation shows fewer verifier-seconds per training step than all-rollout verification, with no loss in final quality.

The oracle-evidence study decides the interpretation. If supplying the true fault location removes most false commits for every method, the failure lies in finding evidence rather than in deciding when to stop, and effort should shift to search. If false commits persist and only hindsight credit reduces them, stopping is confirmed as a separately trainable skill in security repair.