Certificate-Checked Reinforcement Learning for Dismissing Static Analysis Warnings

Static bug finders such as Infer and CodeQL report every code path that might fail. Most of those paths can never actually run, so in large C codebases the large majority of warnings are false alarms, and engineers learn to ignore the tool. A promising recent fix has a language model read the warned path and translate its branch conditions into a logic formula. A solver then checks whether any input could drive execution down that path; if none can, the warning is discarded. This works well at inference time, but it costs many expensive model calls per warning. It also has a deeper weakness. The solver checks only the formula the model wrote, not whether that formula describes the code.

We propose to train a small model with reinforcement learning so that it produces a checkable certificate of infeasibility for each warning it dismisses. Here reinforcement learning means improving a model by rewarding its own sampled outputs. The certificate contains short claims about what each called function returns, plus loop facts. Each claim is verified mechanically against that one function's source code. The path condition itself is extracted by a tool, not written by the model. The model earns reward only when every claim checks out and the claims, combined with the extracted path, are logically contradictory. Under this design, a warning that is a real bug cannot be rewarded for dismissal unless the bounded checker is wrong. Training also needs no human labels, because the checker supplies the reward on any warning a static analyzer emits.

Core Claim

The obvious bridge between the two fields is reinforcement learning with verifiable rewards (RLVR), where the reward comes from an automatic checker rather than a learned judge, applied to the solver's verdict on model-written path constraints. That design is gameable. A model can always write an unsatisfiable formula, the solver will happily answer "no path exists", and the model learns to dismiss everything, real bugs included. Our claim is that the reward must certify the correspondence between the formula and the program, and that this can be done cheaply. Proving a whole inter-procedural path infeasible is hard and runs into path explosion. Checking a single proposed function summary against a single function body is local and bounded. That gap between hard search and cheap checking is the same one exploited by prover–verifier training. We expect a model trained on this reward, using only unlabeled warnings, to approach the false-alarm reduction of agentic inference-time pipelines while keeping real-bug recall near 1 by construction, at a small fraction of their cost.

Relation to Prior Work

Three works motivate the problem and supply the closest system. Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis (LLM4PFA) has prompted agents analyze a path function by function, retrieve callee code, and emit SMT-LIB queries for the Z3 solver. On SAFP-BENCH-C it reduces false positives by 72% to 96% while missing 3 of 45 real bugs. It updates no weights, pays multi-turn inference cost per warning, and trusts model-extracted conditions. Its authors name fine-tuning local models as future work. We differ in three ways: we train the model, we take path extraction away from the model, and we demand verified callee summaries rather than asserted ones. Learning to Reduce False Positives in Analytic Bug Detectors filters Infer warnings with a supervised CodeBERTa classifier or a zero-shot completion proxy. Those filters give a verdict with no checkable reason, and they need developer labels. Efficiently Detecting Reentrancy Vulnerabilities in Complex Smart Contracts and Unity is Strength show that slicing plus targeted symbolic execution can raise precision sharply in smart contracts. They do so with hand-engineered, vulnerability-specific machinery that we do not assume is available for C.

Two works supply the method. Models That Prove Their Own Correctness trains a prover to emit certificates that a fixed checker accepts. Its RL from verifier feedback lifts verifiability on greatest-common-divisor arithmetic from 60.3% to 78.3%, and annotated intermediate steps reach 96.0%. We carry over the answer-plus-certificate design and the lesson that step annotations help warm-start the model, moving from arithmetic to code. Cascade shows that bounded symbolic simulation of a single C function with an SMT back end is fast, which is the kind of per-function checker our reward needs. Solver-grounded RLVR has worked on optimization modeling in Solver-Informed RL and on Z3-verified puzzles in PuzzleClone. In both, the solver's verdict is the ground truth. In our setting the solver's verdict is only meaningful if the formula is faithful to the code, which is the problem we address.

Two works constrain the design. Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning finds that outcome-only RLVR degraded reasoning faithfulness in 19 of 40 tasks. That result argues against rewarding a bare "false positive" verdict. Our design rewards a checked artifact instead, so the free-text reasoning never needs to be trusted. The same paper finds that 64 expert traces can prevent the collapse, which shapes our warm start. Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers corrects policy gradients for verifier noise, assuming constant error rates on math. Our checker's errors are structural rather than constant. Timeouts reject valid summaries, and finite loop unrolling can accept invalid ones. Both kinds of error depend on code shape, which is exactly the extension that paper leaves open.

The Certificate and the Reward

For a warning, the static analyzer supplies a trace through functions $f_1, \dots, f_n$. A tool extracts the intraprocedural branch conditions along that trace and calls the result $\phi_{\text{path}}$. Callee return values and loop effects stay as uninterpreted symbols. The model outputs a set of summaries. Each summary $s_g = (\text{pre}_g, \text{post}_g)$ is attached to a callee or loop $g$. The certificate is accepted when three things hold. First, a bounded model check of $g$ alone, with unroll bound $k$, confirms that $\text{pre}_g$ implies $\text{post}_g$. Second, at each use, Z3 confirms that $\text{pre}_g$ follows from the path context. Third, the combined formula is unsatisfiable:

$$ r = \mathbb{1}\Big[\textstyle\bigwedge_g \mathrm{BMC}_k(g, s_g) \;\wedge\; \bigwedge_g (\phi_{\text{path}} \Rightarrow \text{pre}_g) \;\wedge\; \mathrm{UNSAT}\big(\phi_{\text{path}} \wedge \bigwedge_g \text{post}_g\big)\Big]. $$

Here $\mathrm{BMC}_k(g, s_g)$ is true when the per-function check passes within the time limit. A vacuous precondition such as $\text{pre}_g = \text{false}$ fails the second condition, so it cannot be used to cheat. The trace's entry function may need an assumption about its own arguments. That assumption must be discharged at every call site in the repository, up to depth $d$. Otherwise the certificate is labeled "conditional": it is shown to the auditor but earns no reward. When the model emits no certificate, or the certificate fails, the warning is kept. This failure mode is the safe one.

Research Program

We will build the environment from Infer and CodeQL warnings on open-source C projects that do not overlap with the Linux kernel, OpenSSL, or Libav, which keeps SAFP-BENCH-C uncontaminated. The training set will contain thousands of warnings, none of them labeled. The policy is a 7B-parameter code model of the Qwen2.5-7B class used by the noisy-verifier and PuzzleClone papers.

Without help, the model would almost never find a valid certificate, because random exploration rarely does, as the file's discussion of sparse rewards notes. We therefore warm-start it on a few hundred certificates. These come from running an LLM4PFA-style prompted pipeline with a large model and keeping only the outputs that pass our checker. This follows the annotated-transcript lesson of Amit et al. and the small-SFT finding of Yu et al.

On-policy training then samples several certificates per warning and applies group-normalized policy gradients with the binary reward $r$. Verifier noise is handled in two directions. Timeouts are the false negatives. We re-run a sample of timed-out checks with a ten-times-larger budget to estimate the false-negative rate $\rho_1$ separately for buckets of loop depth and callee size. We then apply forward correction from Cai et al. using these bucketed estimates rather than one constant. Unsound acceptances at bound $k$ are the false positives. We audit a random fraction of accepted summaries at bound $4k$ and penalize any summary that fails the deeper check. This gives an empirical estimate of how often the bounded checker is fooled, and whether training teaches the model to exploit the bound.

Baselines

The naive reference is the raw analyzer output, which keeps every warning. Beside it we run whole-path bounded model checking with no learning. Each SAFP-BENCH-C trace is inlined and checked with the same solver, the same bound $k$, and the same per-warning wall-clock budget our method uses. This tells us how much of the problem plain formal methods already solve. If it already removes most false alarms within budget, learning is not needed.

The strongest existing method is LLM4PFA, which reports a 72% to 96% false-positive reduction recall on SAFP-BENCH-C with 0.93 real-bug recall (3 of 45 real bugs missed), per Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis. We will rerun it twice: once with its original large model, and once with our 7B base model under the same retrieval tool. We will report reduction, recall, tokens, and wall-clock per warning for both runs. Beating it means one of two outcomes at 5% or less of its per-warning cost. Either we match its reduction within 5 points while missing no more real bugs, or we exceed its real-bug recall at equal reduction.

The supervised baseline is a DeepInferEnhance-style classifier from Learning to Reduce False Positives in Analytic Bug Detectors. It is fine-tuned from our base model on the same number of labeled warnings our warm start consumes, with an equal tuning budget. Its reported 85.4% precision is on a Java Infer dataset, so no reported result exists on Lacuna for it on SAFP-BENCH-C, and we measure it ourselves.

Two ablations isolate the key ingredient. The first is the obvious RLVR design: the model writes the whole path formula, and Z3 UNSAT is the reward. It shows what happens when the formula is not tied to code. The second is rejection-sampling fine-tuning: repeated rounds of supervised training on checker-accepted certificates, with the same checker and sample budget as RL. It separates the value of the certificate from the value of the RL algorithm.

Why not just…?

Why not just distill LLM4PFA into a small model with supervised fine-tuning? Distillation would keep LLM4PFA's trust in model-extracted conditions, and the teacher misses real bugs on the benchmark. Copying the teacher's outputs cannot remove an error source the teacher has. Filtering the distilled data with our checker is a real competitor, and it is exactly the rejection-sampling ablation. If that ablation matches full RL, we will report that the checkable certificate is the contribution and the RL algorithm is interchangeable. That would still be a useful result.

Why not just train a classifier on labeled warnings? Labeled warnings number in the hundreds (364 in SAFP-BENCH-C, 539 in Kharkar et al.), each labeled by experts. A classifier's dismissal also gives an auditor nothing to check. The finding by Amit et al. that answers can be 99.8% correct with 0% verifiability illustrates that accuracy and checkability come apart. A classifier might still match our accuracy. We will report this honestly, and argue for our method on recall guarantees and auditability rather than on accuracy alone.

Why not just reward Z3's verdict on the model's own formula? That design rewards exactly the gap between formula and code that our certificate closes, and outcome-only rewards are known to encourage unfaithful reasoning (Yu et al.). The first ablation tests the prediction directly. We expect real-bug recall to fall well below LLM4PFA's 0.93 as training proceeds, while the training reward keeps rising.

Why not just run symbolic execution on the whole path? Whole-path symbolic execution is the strongest formal alternative, and it may succeed on short traces. The path-explosion and timeout limits documented in the literature suggest it will fail on deep kernel call chains. The whole-path baseline, broken down by trace depth, will show where this happens and how often.

Proposed Experiments

The main evaluation is SAFP-BENCH-C. We report false-positive reduction recall and real-bug recall per bug type, along with certificate acceptance rate and per-warning cost.

Three checks are decisive. The first is the hacking test. Over training, the Z3-only ablation should lose real-bug recall while our method stays at or above 0.93. Any real bug that receives an accepted certificate is a soundness failure, and we will analyze each case to see whether the bound $k$ or a modeling gap caused it.

The second is the bounded-checker audit. Among accepted summaries, the fraction that fail at bound $4k$ should stay flat during training. A rise would mean the policy is learning to exploit the bound, and would justify a larger $k$ or the deeper-check penalty.

The third is label-free learning. Reduction recall should rise from the warm-started model to the RL-trained model even though no training warning carries a label, and the gain should hold on Linux kernel warnings, whose heavy macro use stresses the path extractor. We will separately report warnings dropped because extraction failed, since errors in recovering control flow are a known weakness that this design does not remove.

Secondary analyses measure how the bucketed noise estimates compare against a single constant $\rho_1$. They also report how often certificates are conditional rather than closed, because conditional certificates are a natural place to bring a human auditor in.