Language-model theorem provers increasingly work top-down. They first write a proof sketch, an outline whose intermediate claims are left unproved behind a placeholder such as Lean's sorry ("assume this for now"). Then they try to prove each claim separately. The weak point is that a sketch can be accepted by the proof checker while one of its claims is false, or is true only under hypotheses the sketch forgot to carry along. The prover then spends its whole search budget on a goal that cannot be proved. This note proposes a two-role protocol. A disprover agent attacks every intermediate claim of a fresh sketch with a counterexample search whose results are checked by the Lean kernel. The concrete counterexample is returned to the planner, which must repair the claim, for example by adding the missing hypothesis or weakening the conclusion, before any tactic-level search begins.
The core bet is specific and can be tested. A machine-checked counterexample to a subgoal is a far more useful repair signal than "this branch failed after N search steps" (how POETRY finds out) or "the compiler is happy" (how sorry-based pipelines see it). Because the kernel, not a language model, settles every disagreement, the protocol avoids the false-consensus and distraction failures that make ordinary multi-agent debate unreliable. The idea joins three lines of work. Hierarchical proof decomposition supplies the setting. Multi-agent task decomposition supplies the split between a role that generates claims and a role that falsifies them. Structured adversarial debate supplies the rule that every objection must carry evidence. Here, evidence means a kernel-certified witness.
What exists and where the gap is
Several hierarchical provers write sketches and then fill them in. POETRY writes Isabelle sketches with sorry and proves the deferred conjectures by recursive best-first search, so it learns a conjecture is unprovable only when that search is exhausted. ProD-RL trains a model to propose sub-lemmas and rewards any verified subtree. APOLLO repairs failed proofs by replacing failing blocks with sorry and re-invoking the model locally, and its authors note it cannot recover when the high-level structure is unsound. The closest work is SketchRefine. It already sorts failed steps into "logically incorrect" and "computationally hard," and it has a refutation stage. That refutation tries to derive False from the problem's top-level assumptions, so it catches bad theorem statements, not bad intermediate claims. Its failure sorting is also done by the model, after proving has been attempted. All four motivate this proposal: each either finds false subgoals late, or finds them without a certificate, or does not find them at all.
The method comes from two places. Lean Disprove couples a model-driven search loop with SAT/SMT solvers, enumeration and Lean decision procedures. It returns REFUTED only when the kernel accepts the witness. On its 16-conjecture diagnostic set it certified all 15 false targets and correctly returned INCONCLUSIVE on the true control. It has only been run on standalone conjectures, never inside a prover's loop. FormalML provides the to_theorem tactic, which turns a proof step together with its local context into a standalone Lean statement. That is exactly the object the disprover should attack.
Two results from the debate literature constrain the design. Adversarial Review shows that unconstrained agent conversation drifts into false consensus. Requiring objections to cite the artifact (DISAGREE_EVIDENCE) fixes this, and its three-agent design beat a five-agent round table. FORD shows the opposite failure: a weaker, stubborn agent can talk a stronger one out of a correct answer. Both failures come from letting a language model referee the dispute. We adopt Adversarial Review's evidence requirement and make it strict. A refutation counts only if the kernel accepts it, and an uncertified objection cannot block a claim; it can only lower that claim's priority. MA-LoT also splits proving into two roles, a Prover and a compiler-error Corrector. But a sorry-sketch with a false claim produces no compiler error, so its Corrector never fires on the failure studied here.
Core Claim
Consider a planner that proposes a sketch with subgoals $g_1,\dots,g_k$, each turned by to_theorem into a closed statement that includes its local hypotheses. A cheap certified falsification pass over every $g_i$, followed by repair conditioned on the witness, will produce two effects. It will solve more theorems at a fixed budget of Lean calls and model tokens than either uninformed resampling of the sketch or late, search-based discovery of unprovable subgoals. The gain should be concentrated on problems whose failed sketches contain false subgoals, so it should grow on deeper undergraduate and research-style material (PutnamBench, FormalML, FormalMath-Lite) relative to shallow competition problems. This direction follows from ProD-RL's report that decomposition helped on library proofs but slightly hurt on miniF2F.
Research Program
The first stage is a diagnostic study, because the whole project depends on its answer. We collect failed sketches from an APOLLO-style pipeline and a SketchRefine-style pipeline, both run with the same open-weight prover, on FormalMath-Lite, FormalML and PutnamBench. Every unproved subgoal is extracted with to_theorem and given a long certified-disproof search. This yields the fraction of sketch failures caused by subgoals that are certifiably false in their local context, broken down by benchmark and proof depth. If that fraction is small, say a few percent, the false-sketch trap is not the bottleneck and we stop. What remains is a useful negative result about where hierarchical provers actually fail.
The second stage builds the protocol. The planner writes a sketch. The disprover receives each extracted subgoal and runs a budgeted Lean-Disprove-style loop. Its search tools are decide, bounded enumeration, an SMT solver and small model-written search scripts, and its outputs have the same three states as Lean Disprove (refuted, uncertified witness, inconclusive). The planner sees only certified refutations, presented as a short message: the subgoal, the concrete witness, and which hypotheses the witness satisfies. Its only allowed responses are to add hypotheses, weaken the conclusion, replace the subgoal with a different decomposition, or abandon the sketch. A repaired subgoal goes back to the disprover, so the dialogue is a bounded series of kernel-checked exchanges, not open conversation. Subgoals that survive go to the existing tactic-level filler (APOLLO's auto-solver plus local re-invocation), so the proposal changes only what enters the search.
The third stage studies cost. The disprover budget per subgoal, $b$, trades against the proving budget. We report pass rate as a function of total Lean calls, which avoids the token inflation FormalML documents for long-reasoning provers. The planner can reason at length, while the disprover and the subgoal filler are kept short. This assignment is a direct, though partial, test of the planning-versus-execution tension between MA-LoT and FormalML.
Baselines
The naive reference point is the same open prover generating whole proofs and filtered by the compiler at matched sample count. It sets the floor and checks that the shared prover is not degenerate.
The strongest existing method under matched conditions is APOLLO, run with the identical prover model, Lean version and total budget of Lean calls. Its reported number is 84.9% on miniF2F-test for models under 8B parameters (APOLLO), so we first reproduce it near that value on miniF2F to validate our harness. Its reported PutnamBench and ProofNet scores are not given in the file, so on those benchmarks the comparison is against our own matched re-run. On FormalMath-Lite the best reported number is SketchRefine's 76.0% with Gemini 3.0 Pro (SketchRefine). We will not claim to beat that with a smaller model. Instead we re-implement SketchRefine's model-judged validation stage with our prover and compare at equal budget. miniF2F-test is reported only as a sanity check, since SketchRefine's 99.6% leaves no room to measure an improvement there.
On FormalML the reference is STP's Pass@32 of 63.21%, with best models below 35% on Level 5 (FormalML). Beating the baseline means a statistically significant gain at equal Lean-call budget on at least two of FormalMath-Lite, FormalML Level 4–5 and PutnamBench, measured with paired bootstrap over problems.
Three ablations isolate the key ingredient. The filter-only ablation discards and resamples any sketch with a refuted subgoal, without showing the witness to the planner. It tests whether the counterexample content matters or only the early pruning. The uncertified-judge ablation replaces the kernel-checked disprover with a model that merely argues a subgoal is false, in the style of the MACM Judge or FORD. It tests whether kernel arbitration is what prevents sycophantic or distracting outcomes. The late-discovery ablation runs the same disprover only after a subgoal's proof search fails, as a stand-in for POETRY-style backtracking. It tests whether timing matters.
Why not just…?
Why not just sample more sketches and let the compiler filter? APOLLO shows that structure already cuts Goedel-Prover-SFT's budget from 25,600 samples to a few hundred, so brute force is a real but expensive competitor. The trouble is that a sorry-sketch with a false subgoal passes the compiler. Resampling discovers the problem only by failing to fill the sketch in. The budget-matched curves against the naive baseline and against filter-only resampling measure directly how much the witness feedback adds beyond more draws.
Why not just run Lean Disprove on each subgoal and drop refuted sketches? This is the most serious threat, and it is exactly the filter-only ablation. If filtering alone matches the full protocol, the honest conclusion is that a certified falsification check belongs in hierarchical provers, and the repair dialogue is unnecessary. We would report the result that way. Our expectation that repair helps rests on a prediction we can check: a witness often points to one missing side condition, such as $n=0$ or a non-strict inequality, which is cheap to add but unlikely to appear by chance in a resample.
Why not just let the model sort failures, as SketchRefine does? That sorting happens after a proof attempt and has no certificate, so a confident but wrong "computationally hard" verdict sends more search at an impossible goal. The uncertified-judge ablation and the matched SketchRefine re-implementation compare model judgment against kernel-checked refutation on the same subgoals.
Why not just use built-in decision procedures? We do use them inside the disprover. But on Lean Disprove's diagnostic set, bare decide caught 10 of 15 false targets against 15 for the agentic loop, so tactics alone miss the nonlinear and out-of-window cases. We will report recall for decide alone against the full disprover on the planted-error set described next.
Proposed Experiments
The decisive diagnostic is the false-subgoal fraction from the first stage, reported for each benchmark. To measure detection without relying on naturally occurring errors, we build a planted-error set. We take verified FormalML and FormalMath-Lite subgoals, extract them with to_theorem, and mutate them by dropping a hypothesis, flipping a strict inequality, or shifting a bound. Mutants that are certifiably false give known ground truth, and mutants the disprover cannot resolve are kept and labeled unknown. On this set we report disprover recall at each budget $b$. We also check that the false-refutation rate on unmutated true subgoals is exactly zero; anything else means the certification code has a bug.
The main experiment compares the full protocol, the three ablations, the naive baseline and the matched APOLLO and SketchRefine re-runs. It uses FormalMath-Lite, FormalML stratified by difficulty level, PutnamBench in Lean 4, and miniF2F-test as a sanity check. The primary metric is pass rate at fixed Lean-call budget. Secondary metrics are search spent on subgoals later shown to be false, the number of repair rounds per solved problem, and the share of final proofs whose sketch was repaired after a refutation.
The headline prediction is about where the gain lands. Improvement over APOLLO should be largest on problems whose failed sketches contained certified-false subgoals, and near zero elsewhere. If the gain is uniform across problems, or appears without any refutations firing, the mechanism claimed here is not what produced it, and the note's central claim fails.
References
- Proving Theorems Recursively
- Formal Theorem Proving by Rewarding LLMs to Decompose Proofs Hierarchically
- APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning
- Editable Proof Sketch for Automated Theorem Proving
- Lean Disprove: Certified Counterexample Search for AI-Assisted Formal Mathematics
- FormalML: A Benchmark for Evaluating Formal Subgoal Completion in Machine Learning Theory
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
- MA-LoT: Model-Collaboration Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Proving