When a language model translates a word problem into formal logic and hands it to a theorem prover, the prover does the reasoning. What it cannot catch is a correctly formatted but wrong translation. One mistranslated premise, such as a flipped quantifier or a dropped negation, lets the prover derive a confident wrong answer. Current systems repair translations by looping: run the prover, paste its complaint back into the prompt, ask the model to try again. Each turn, the prompt grows. It carries the full problem, every earlier failed attempt, and raw prover logs. Research on distracting context shows that models get worse as irrelevant text piles up. Research on iterative coding agents shows their code degrades with every revision.
We propose to let the prover decide what the model sees during repair. First, the model samples several translations. The prover then tells us two things. It tells us which premises each proof actually used (its "support"), and on which formal premises the samples disagree. Where these two sets overlap, we have a small set of suspect premises. Each repair turn then shows the model only those natural-language sentences, their competing formalizations, and the definitions they share. It sees no history of failed attempts and no untouched premises. The bet is that a context window chosen per failure, using signals from the prover, repairs meaning errors more reliably and more cheaply than the growing transcripts that current self-refiners use.
Core Claim
In neuro-symbolic reasoning, the model does the translation and a deterministic solver does the deduction. The main loss is at the translation step. The main tool against it is iterative repair, and that tool is weakened by its own context: irrelevant premises and accumulated failed attempts. We claim three things.
First, repair accuracy under full-history prompting falls measurably as the turn count and the number of distractor premises grow.
Second, the solver's proof support, intersected with disagreement across sampled translations, identifies mistranslated premises with useful precision. We can check this directly against FOLIO's expert first-order logic annotations.
Third, a repair loop that rebuilds a minimal, fresh context each turn from that suspect set does better than full-history refinement, generic compression, and plain resampling when all methods get the same token budget.
The third claim is the risky one, and the experiments are built to break it.
How This Sits Relative to Prior Work
Some prior work shows the gap exists. Logic-LM pairs a model with Pyke, Z3 and Prover9 and adds a self-refiner that feeds solver error messages back to the model. That refiner acts only on syntax errors. It translates the whole prompt at once, and its prompt grows across turns. We keep its solver interface but target meaning errors that execute silently, and we replace the growing prompt with a rebuilt one. FOLIO reports that Logic-LM with GPT-4 reaches 78.1%, against 96% for human experts. That gap is our target, and FOLIO's expert annotations make localization measurable. SlopCodeBench finds that coding agents given their full cumulative history erode code quality in 77.0% of trajectories. It proposes no fix and never tests formal languages. We ask whether the same decay appears in logic translation, and whether controlling context arrests it. Less is More shows that narrative filler in programming problems misleads code generation, and that summarizing problems helps only in a single pass. It also finds only about 2% gains on the hardest tier. That is a warning that stripping noise cannot substitute for reasoning, and a reason our pruning is driven by feedback rather than done once up front.
Other work supplies methods we reuse. LoC-Decomp back-translates formal segments into natural language to check whether they mean the right thing. It appends critiques across iterations without pruning, and it names retrieval as future work. We use back-translation only as a tie-breaker inside the suspect set. Decompose, Structure, and Repair maps Lean compiler error locations onto branches of an operator tree so that repairs stay local. That localization depends on a compiler reporting where the error is. Meaning errors in first-order logic have no such location, and the prover's proof support is the substitute we propose. COCOGEN uses compiler errors as retrieval queries to fetch missing definitions. Its context only grows. Ours both adds (shared predicate definitions) and removes (everything else).
A third group constrains the design. DynamicRAG learns a variable-length document list from a reward on the final answer. Its authors note that it lacks step-wise signals and was tested only on question answering. The solver gives us exactly such a step-wise signal without training a reranker. EXIT filters sentences with a small classifier. Its sentence-by-sentence selection can cut links between premises, which is a known risk for logic. This is why our window always includes premises that share predicates with the suspect set. MINIMAL shows that most reading-comprehension questions need only one or two sentences, which suggests minimal windows are plausible. Its selector scores each sentence on its own and cannot see across the proof. IFDNS reorders premises to help deduction but never discards any, and it reports heavy token cost. Our method should cut tokens rather than add them. Improving Symbolic Translation inserts a check on predicate arity (the number of arguments each predicate takes) between predicate generation and translation. That check is purely structural and open-loop. We adopt its two-stage predicate-first format as the translation template.
Research Program
The system has three parts.
The translator samples $K$ full translations of the premises and conclusion into first-order logic, predicate declarations first.
The localizer runs the prover on each sample. It records the answer (true, false, or unknown) and the set of premises used in each proof, taken from proof objects or, for refutations, from unsat cores. An unsat core is the smallest subset of formulas that the solver found contradictory. It also computes, for each premise $i$, whether its formalizations disagree across samples. We check this by equivalence under Z3, not by string match. The suspect set at turn $t$ is
$$W_t = \Big(\bigcup_{k} S_k\Big) \cap D \;\cup\; N(\cdot),$$
where $S_k$ is the premise support of sample $k$'s proof, $D$ is the set of premises with non-equivalent formalizations, and $N(\cdot)$ adds any premise that shares a predicate with a member of the intersection. If samples disagree on the answer but $D$ is empty, the problem lies in the predicate vocabulary, so the window becomes the predicate declarations.
The repairer gets a fresh prompt. It contains the sentences in $W_t$, their competing formalizations, the shared predicate declarations, and a one-line statement of the conflict (for example, "samples 1 and 3 prove the conclusion via premises 4 and 7; sample 2 cannot"). It chooses or rewrites one formalization per sentence, and those are spliced back into the full program. Following MEIC, we keep the best stable state and roll back any repair that breaks previously consistent premises. The loop stops when the samples agree or when a turn budget runs out.
The analysis side measures a "translation erosion" rate. This is the fraction of repair turns that alter a premise which was correct before the turn, scored against FOLIO gold annotations. It is the logic analogue of SlopCodeBench's erosion metric. We plot it against turn number for every method.
Baselines
The naive reference point is single-pass translation plus solver with no repair. This is Logic-LM's pipeline with its self-refiner switched off, run with our models and prompts. Any method must beat it to justify extra tokens.
The strongest existing method is Logic-LM with its self-refiner. For fairness we extend it past syntax errors: when sampled answers disagree, the full-history prompt also receives the disagreement. That gives it the same trigger signal as our method and isolates context construction as the only difference. For reference, the file reports FOLIO's figure of 78.1% accuracy for Logic-LM with GPT-4, and Logic-LM's GPT-4 improvement on AR-LSAT from roughly 35% to over 43%. We rerun Logic-LM ourselves on each backbone rather than compare to those numbers directly. Beating it means higher accuracy at equal or lower total tokens, with non-overlapping bootstrap intervals on FOLIO and AR-LSAT.
The third baseline is self-consistency over translations: majority vote over the solver answers of $N$ sampled translations, with $N$ chosen to match our total token spend. The fourth is generic compression. It is the full-history refiner with history and premises compressed by an EXIT-style sentence classifier, or alternatively by an LLM summary. This tests whether any shortening helps or whether solver-chosen shortening is what matters.
Two ablations isolate the key ingredient. Fresh context without localization rebuilds the prompt each turn with all premises but no history. It separates removing history from removing premises. Random window of matched size keeps the prompt size of $W_t$ but picks premises at random. If our method does not clearly beat both ablations, the localizer adds nothing. All methods share backbone models, the sampling temperature, a per-problem token cap, and an equal prompt-tuning budget on a held-out split of ProofWriter.
Why not just…?
Why not just resample from scratch until the samples agree? This is the most serious threat. Plain resampling already enjoys a fresh context, and its gains are hard to beat when errors are independent across samples. Our argument is that mistranslations are correlated: the same ambiguous sentence misleads most samples, so voting locks in the error. Showing the model the competing readings of that one sentence should break the tie. The self-consistency baseline at matched tokens settles this directly. If it matches us, we will report that the localizer helps diagnosis but not accuracy, and reposition the contribution as cheaper diagnosis.
Why not just use a long-context model and ignore the noise? Stronger models may tolerate distractors. The evidence we have points the other way. Improving Zero-shot Reader needed explicit distraction handling to recover large exact-match losses under 100 distracting documents. SlopCodeBench shows that degradation over turns persists in current agents. Our noise-injection sweep, run on the strongest API model available, tests this directly. A flat curve for full-history refinement on that model would limit our claim to smaller models.
Why not just localize with back-translation, as LoC-Decomp does? Back-translation checks every segment and costs a model call per segment. It also relies on the model's judgment of whether two paraphrases agree. The prover's support set is free and exact. We compare localization precision and recall against FOLIO gold annotations for back-translation alone, solver signals alone, and their combination, reporting token cost for each.
Why not just prune irrelevant premises once, before translation? One-time pruning is what Less is More and EXIT do. It risks discarding a premise that turns out to be needed, and a premise's relevance to a proof is often only visible after proving. The one-time-pruning variant appears among the compression baselines. We also report how often it drops a premise that the gold proof uses.
Proposed Experiments
The main benchmarks are FOLIO (using its expert annotations for localization scoring), ProofWriter at depths 0 to 5, PrOntoQA, LogicalDeduction and AR-LSAT, the same suite as Logic-LM. We run two backbones: an open 8B instruction model and a stronger API model. The prover is Prover9 or Z3, depending on the dataset, following Logic-LM.
To probe noise, we build distractor variants of FOLIO and ProofWriter. Each adds $m \in \{0, 5, 10, 20\}$ premises that use the same predicates but do not affect the gold proof. ProofWriter's generator makes such premises natural. Metrics are answer accuracy, total tokens per problem, localization precision and recall, and translation erosion per turn.
As an exploratory transfer study, we swap the prover for the Lean compiler and apply the same localizer, using disagreement across samples and error locations, on ProofNet. There, LoC-Decomp reports a 73.57% pass rate and Decompose, Structure, and Repair reports a 79.51% consistency-check pass rate. These are context for the transfer study, not targets we promise to beat.
Three checks decide the project. First, under full-history refinement, accuracy and erosion must worsen with turn count and with $m$. If they do not, the premise that context noise hurts repair is false for this setting and the project stops. Second, on FOLIO gold annotations, the suspect set must beat a random window of equal size at localization by a wide margin. Otherwise the prover's signals are not informative. Third, at matched tokens, our loop must beat both Logic-LM's extended self-refiner and self-consistency, with the advantage growing as $m$ grows. A method that wins only at $m=0$, or only against weak baselines, does not support the claim.
References
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
- FOLIO: Natural Language Reasoning with First-Order Logic
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
- Less is More: Summary of Long Instructions is Better for Program Synthesis
- LoC-Decomp: LLM Autoformalization via Logical Concept Decomposition and Iterative Feedback Correction
- Decompose, Structure, and Repair: A Neuro-Symbolic Framework for Autoformalization via Operator Trees
- Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation
- EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation
- Efficient and Robust Question Answering from Minimal Context over Documents
- IFDNS: An Iterative Feedback-Driven Neuro-Symbolic Method for Faithful Logical Reasoning
- Improving Symbolic Translation of Language Models for Logical Reasoning
- MEIC: Re-thinking RTL Debug Automation using LLMs
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering