When a language model writes scientific code that fails its tests, today's repair loops show it the error message and its own broken program and ask for a fix. That setup quietly invites confirmation bias. The model's first program acts as an anchor: the model assumes its approach and its domain assumptions are right, and it patches whatever line the traceback points to. In expert domains the real cause is often somewhere else, such as a wrong physical convention, a misremembered theorem, or an algorithm that is wrong for the problem. We propose to change the step between "the test failed" and "here is the patch." Before editing anything, the model must write down several competing explanations for the failure. Each explanation must name one cheap, runnable probe that could rule it out, and, when the explanation involves domain knowledge, a targeted query to a reference corpus. The sandbox runs the probes, and the surviving explanation decides what gets retrieved and what gets repaired. Execution picks the diagnosis. The model's own judgment of its reasoning does not.
This matters because the gains from iterative repair stall early exactly where they are most needed. On SciCode, a benchmark of research-grade scientific coding, the best reported model solves only 7.7% of main problems (SciCode). Self-debugging on code translation adds a few points and then flattens out (CodeTransOcean). Sixty-four rounds of compiler feedback in Lean leave models going in circles (VERINA). Our bet is that a large share of these stalls are diagnosis failures caused by the model anchoring on its first attempt. We think they are not simply missing knowledge, and that forcing the model to set up rival explanations and then settle them by execution recovers much of the loss.
Core Claim
The claim has two parts. First, in execution-guided repair of domain code, how often the fix works depends mainly on whether the model identifies the correct category of cause: a local coding error, a wrong domain principle, or a wrong algorithmic strategy. Unprompted repair loops identify the local-coding category far more often than the true distribution warrants. Second, the reasoning-based debiasing that works for judges and classifiers carries over to repair, but only if each hypothesis is tied to an executable check. Verbal justification alone is not enough. That condition is not an extra we added for convenience. The literature says it is required. Chain-of-thought rationales often do not reflect what actually drove the answer (Language Models Don't Always Say What They Think), and bias in a model's written reasoning correlates only weakly with bias in its outputs (Do Biased Models Have Biased Thoughts?). A rival hypothesis that is only argued in words can be argued away. One that must survive a probe cannot.
Where This Sits
The method borrows from two debiasing results and addresses a gap in two agentic systems. From Think Twice Before Trusting ($T^3$) we take the core move. That paper removes confirmation bias in self-evaluation by making the model justify every candidate answer before scoring confidence, but it only works over small, discrete label sets and draws only on parametric memory. We replace the label set with a set of failure explanations and replace the model's confidence score with probe outcomes. From Any Large Language Model Can Be a Reliable Judge (RBD) we take the finding that a structured account of why a verdict is wrong corrects a judge where a bare flag does not. That paper also reports that stronger models resist correction more. We treat this as a limit on design: the corrective signal has to be something the model cannot talk its way around, which is why we use execution. Do LLM Evaluators Prefer Themselves for a Reason? motivates the work. It shows harmful self-preference reaching 86% on MATH500 for a strong model, and it also shows that long-reasoning models reduce this bias. That second result is a competing explanation we have to test directly (see below).
On the repair side, ARCS is the closest existing system. It retrieves code and documentation using subgoals derived from the task, then repairs from tracebacks under a budget while keeping the best candidate so far. Its retrieval happens once, is driven by the task statement, and never asks whether the model's own assumptions are the problem. In our method retrieval is driven by the failure: each rival hypothesis carries its own query, and only surviving hypotheses get their evidence into the repair prompt. PaperRepro recovers unwritten domain conventions from citation-neighbor repositories, but it depends on those neighbors existing and does not arbitrate between conflicting patterns. Our probes supply that arbitration. MEIC adds rollback when a fix causes regressions. We keep rollback as part of every baseline, because it guards against bad patches but does nothing about bad diagnoses. RaDIO triggers retrieval from hidden-state hallucination detectors in open-domain QA. We trigger retrieval from an execution failure instead, which is observable and does not need a trained probe for each model. TheoremQA supplies both a benchmark and a warning. Handing the model theorem definitions barely helps, because the model cannot connect them to the specific problem. We expect retrieval to help only when it is conditioned on a hypothesis the probes have already singled out, and we test that directly.
Research Program
We will build a repair loop with four stages per round. In the first stage, given the failing program, the test output, and the task, the model writes $k$ explanations (default $k=4$). At least one must come from each of three required categories: local implementation error, wrong domain principle or convention, and wrong algorithmic strategy. Each explanation states a prediction that distinguishes it from the others. Examples: "if the sign convention is flipped, the energy at the known limit will have the opposite sign," or "if the integrator is unstable, halving the step changes the output by more than X." Each explanation also gives a short probe script and, where relevant, a retrieval query. In the second stage the sandbox runs all probes. The model does not see the probe code while interpreting the results. It sees only each hypothesis's stated prediction next to the observed outcome, so it cannot reinterpret the probe after the fact. In the third stage, retrieval runs only for surviving hypotheses. If none survive, the loop falls back to strategy-level regeneration. In the fourth stage the repair prompt contains the surviving hypothesis, its probe evidence, and its retrieved passages, and it explicitly allows rewriting from scratch when the surviving hypothesis concerns strategy. MEIC-style rollback and a best-so-far tracker apply throughout.
The retrieval corpus for SciCode will be a fixed collection of reference text covering its subfields. For TheoremQA it will be the dataset's own theorem inventory, so that our condition can be compared directly with its Theorem Augmentation result. For RTL debugging it will be Verilog language and simulator documentation. We will use the same corpus, retriever, and chunking for every retrieval-using condition.
The analytical core is a diagnosis benchmark built by planting bugs with known ground-truth causes into the reference solutions of SciCode subproblems and TheoremQA program-of-thought solutions. There are three bug families matching the three categories, for example off-by-one indexing, a swapped physical convention or wrong constant, and an integrator or solver unsuitable for the regime. Because the true cause is known, we can measure directly what outcome metrics only hint at. We measure diagnostic accuracy, meaning the fraction of rounds whose chosen explanation matches the planted category. We also measure an anchoring rate: the fraction of repairs that keep the failing program's algorithmic skeleton when the planted cause was strategic, operationalized by an edit-distance threshold on the abstract syntax tree plus manual audit of a sample.
Baselines
The naive reference point is a single generation with no repair, reported both for our backbone models and through the numbers the file gives: SciCode main-problem pass rates of 7.7% for o1-preview and 4.6% for Claude 3.5 Sonnet (SciCode), and 51.0% program-of-thought accuracy for GPT-4 on TheoremQA (TheoremQA). We will rerun single-shot with our own backbones instead of comparing across setups.
The second baseline is plain traceback self-debugging with rollback: the loop feeds back the error and the prior code, and reverts regressions as in MEIC. This is the current default, and the evidence that it plateaus comes from CodeTransOcean (48.57% rising only to 52.57% after three rounds) and VERINA (about 20% proof success after 64 rounds).
The strongest existing method is ARCS (ARCS), reimplemented with our backbone, corpus, and retriever. It is the strongest competitor because it already combines domain retrieval, sandboxed repair, and budget control. Its reported results are 87.2% pass@1 on HumanEval with a Llama-3.1-405B backbone and a CodeBLEU gain of +0.115 over standard RAG on the LANL scientific corpus. No reported result was found on Lacuna for ARCS on SciCode, TheoremQA, or the RTL bug benchmark, so all ARCS comparisons will use our reimplementation. On RTL debugging, the reference is MEIC's reported 78% functional fix rate with GPT-4, against a 66% standard-RAG baseline.
Fairness is enforced by a single budget per task, counted as total generated tokens plus sandbox executions, and applied to every method. Probe executions count against our method's budget. Each method gets the same prompt-tuning allowance on a held-out development split. In SciCode, the repair loop sees results only from a visible portion of each subproblem's tests, and scoring uses the held-out portion, so no method gains from overfitting to the tests it is scored on. Beating ARCS means a statistically significant gain in SciCode main-problem pass rate at matched budget, using a paired bootstrap over problems, together with higher diagnostic accuracy on the planted-bug set. A win on only one of these would be reported as partial.
Three ablations isolate the key ingredient. The verbal-only variant keeps the rival explanations but removes the probes, so the model picks a hypothesis by argument alone. This is the direct transfer of $T^3$, and it tests whether execution is necessary. The no-retrieval variant keeps the probes but drops hypothesis-conditioned retrieval. The unconditioned-retrieval variant retrieves using the task statement, as ARCS does, instead of the surviving hypothesis.
Why not just…?
Why not just restart from scratch with fresh samples instead of repairing? This is the most serious threat. Throwing away the failed program removes the anchor completely, and best-of-N sampling with test filtering is cheap. We will run it at exactly matched budget: independent samples plus test-based selection, and also restarts that are given only the failing test names without the prior code. If fresh restarts match our method on SciCode, the claim that structured diagnosis adds value is refuted for that regime, and we will report it that way. We expect our method to win specifically on the domain-principle bug family, because a restart re-samples the same wrong parametric belief, while a surviving convention hypothesis triggers retrieval that corrects it. The planted-bug breakdown is what settles this question.
Why not just use a long-reasoning model, since that already reduces self-preference? Do LLM Evaluators Prefer Themselves for a Reason? does report that long reasoning chains sharply reduce harmful self-preference, so this is a genuine alternative. That evidence, however, comes from single-turn pairwise judging. The same line of work, and RBD, find that stronger models are more stubborn once committed, and the VERINA plateau was observed with frontier models. We will run every condition with a long-reasoning backbone as well as a standard one. If the gap between our method and plain self-debugging disappears under the long-reasoning backbone, then the debiasing is already internalized, and the contribution shrinks to the retrieval component.
Why not just retrieve domain references up front and append them? TheoremQA found that supplying theorem definitions barely helps. CoDA-Bench found that agents latch onto plausible distractors in roughly 20% of tasks, while oracle data raises accuracy by 25–28%. Static retrieval is included as the unconditioned-retrieval ablation. Our prediction is that hypothesis-conditioned retrieval moves closer to the oracle gain because the probes filter what gets retrieved. If it does not, retrieval is not the bottleneck, and we will say so.
Why not just prompt the repair model to "consider that your approach may be wrong"? RBD reports that structured reasoning critiques beat in-context debiasing prompts by 12.8% for judges, and Language Models Don't Always Say What They Think shows that models rationalize around such cues without acknowledging them. We will include this one-line instruction as a cheap baseline, and the verbal-only ablation is its structured version.
Proposed Experiments and Decisive Checks
The main evaluation covers SciCode main problems and subproblems, TheoremQA in program-of-thought form, and the 178-bug RTL benchmark from MEIC. There are two backbones, one standard and one long-reasoning, and three budget levels, so we can see whether our advantage grows or shrinks as budget increases. We report pass rate, or fix rate for RTL, plus diagnostic accuracy and anchoring rate on the planted-bug set.
Three results decide whether the idea holds. First, on planted strategy bugs, plain self-debugging should show a high anchoring rate and our method a markedly lower one. If anchoring is already low for the baselines, the premise is wrong. Second, the full method must beat the verbal-only ablation on diagnostic accuracy. If it does not, execution-grounding is not the ingredient that matters, and the work reduces to a $T^3$ port. Third, at matched budget, the full method must beat fresh-restart sampling on the domain-principle family. If it fails that test, the honest conclusion is that confirmation bias in repair is best handled by discarding history rather than by reasoning about it.
As a secondary stress test, we will run the loop on VERINA proof generation, where the rival explanations are candidate proof strategies and the probes are Lean checks of intermediate lemmas. This asks whether the roughly 20% plateau after 64 rounds reflects anchoring on a single proof route.
References
- SciCode: A Research Coding Benchmark Curated by Scientists
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- VERINA: Benchmarking Verifiable Code Generation
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- Do Biased Models Have Biased Thoughts?
- Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector
- Do LLM Evaluators Prefer Themselves for a Reason?
- ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement
- What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction
- MEIC: Re-thinking RTL Debug Automation using LLMs
- RaDIO: Real-Time Hallucination Detection with Contextual Index Optimized Query Formulation for Dynamic Retrieval Augmented Generation
- TheoremQA: A Theorem-driven Question Answering Dataset
- CoDA-Bench: Can Code Agents Handle Data-Intensive Tasks?