Can Coding Agents Read Their Own Numerical Errors?

When a language model writes a program to solve a physics problem, runs it, and gets the wrong number, the harness usually feeds the wrong number back and asks for a fix. This note asks whether the model actually uses that number. A result like expected 0.00312, got 3.12 carries a diagnosis: a factor of exactly 1000 almost always means a unit prefix was dropped. A factor of 9.81 suggests a missing gravitational constant, and a factor of 2 suggests a lost one-half. Standard subword tokenizers break numbers into chunks by corpus frequency. 0.00312 and 3.12 therefore arrive as unrelated fragments, and nothing in the input lines up their digits or exposes the ratio between them. We suspect that much of the reported gain from execution feedback on numerical code comes from resampling a new attempt, not from diagnosing the mismatch. We also suspect that making numeric feedback readable is what turns the loop into real debugging.

The proposal builds a repair harness on parametric physics problems in which every problem can be re-instantiated with fresh numbers. This does two jobs. It makes memorized answers useless, and it lets us score a program on hidden parameter values the model never saw. On top of the harness we run counterfactual feedback controls: true numbers, a bare "wrong" signal, and numbers taken from a different instance. These measure how much information the model extracts from feedback. We then test three ways to make that feedback readable. The harness can compute the diagnostic itself, the numbers can be re-rendered as aligned digits, or the model can use a digit-level tokenizer.

Core Claim

Our claim is that on numerical scientific code, the benefit of execution feedback is limited by how legible the numbers are, not by the number of refinement rounds. It is measurable as the gap between true feedback and a bare "wrong" signal under matched compute. We predict three things. With raw subword-rendered feedback, that gap will be small. It will be concentrated almost entirely in syntax and exception errors, not in value mismatches. It will widen sharply when numeric discrepancies are made structurally explicit, especially for unit-prefix and physical-constant errors, whose ratios have a recognizable signature.

The claim is falsifiable at each level. If harness-computed ratios in plain text close the gap as well as a digit-level tokenizer does, the lesson is that harnesses should do the arithmetic, and tokenizer changes are unnecessary for this use. That is a useful negative result for the tokenization literature, and we will report it as such.

Where This Sits

Three results motivate the proposal. SymPyBench generates university physics problems from symbolic templates with executable SymPy/Pint code. It finds a "calculation gap": models get about 95% on symbolic forms but drop on numeric counterparts, with roughly 50% confusion across variants of the same problem. It evaluates only one-shot answers, and no model is allowed to run and repair code. TheoremQA shows that moving computation into Python (Program-of-Thought) reaches 51.0% with GPT-4 on expert STEM questions. It also shows that pasting theorem definitions into the prompt does not help much. Its programs are still generated once, with no feedback. Are Non-BPE Tokenizers Ready to Improve LLM Numeracy? shows that distilling a subword model into a single-digit student raises OLMo-3-7B pure-arithmetic accuracy from 27% to 83% on TARITHEVAL Tier 1. It tests only isolated arithmetic, never numbers that appear inside program output.

Two lines of work supply method. OpenCodeInterpreter provides open multi-turn models trained to consume execution traces. We reuse it as the refinement backbone. ICER shows that a relaxed acceptance policy in iterative refinement avoids overfitting to noisy measurements. We adopt its policy so that our comparison is not confounded by acceptance rules. Regression Language Models for Code encodes execution metrics as sign, exponent, and mantissa tokens so that a small model can regress across scales. We borrow that encoding as one feedback rendering. That work predicts metrics offline and never closes a repair loop, whereas we put the encoding on the input side of a loop.

Four results constrain the design. VERINA finds that 64 rounds of Lean compiler feedback lift proof success only to about 20%, with models thrashing. Yan et al. report that feedback gains plateau after 2 to 4 rounds. For these reasons we cap rounds at a small number and compare against resampling at equal token cost, not against single-shot. Rethinking the effects of data contamination in Code Intelligence shows that paired contamination inflates zero-shot scores by 13.41% BLEU but is neutralized by downstream fine-tuning. Our fine-tuned arms therefore need their own contamination check, not an assumption that they inherit one. MLAgentBench shows agents going from 100% on classic tasks to a 37.5% aggregate that includes post-cutoff tasks. This is why the evaluation uses fresh parameters rather than canonical textbook numbers.

Research Program

The harness takes SymPyBench problems whose generating code reproduces the ground truth, the roughly 88% retained by that paper's filter. The model receives the natural-language problem with one parameter instance and writes a Python function of the parameters. The harness runs the function on a small set of visible instances and returns feedback. Final scoring runs the last program on hidden instances drawn from the same template. The main metric is variant-consistent accuracy: the fraction of problems where the final program is correct within tolerance on every hidden instance. Hidden-instance scoring defeats two shortcuts. A program cannot hardcode a remembered answer, and it cannot overfit the visible instance.

Feedback content has three conditions. With true feedback, the model sees expected and obtained values for the visible instances. With null feedback, it sees only pass or fail. With shuffled feedback, it sees correctly formatted expected and obtained values that belong to a different problem. The feedback-use gap is true minus null accuracy at matched token budget. True minus shuffled separates information use from the prompting effect of seeing any numbers.

Feedback rendering has four conditions, applied to true feedback. The first is raw printing, as existing harnesses do. The second re-renders digits with separators so that each digit is its own token, which needs no retraining. The third normalizes to sign/exponent/mantissa in the style of the regression language model and aligns the two values. The fourth is harness-computed diagnostics: the ratio, its base-10 logarithm, and a check against a fixed list of physical constants and SI prefixes, written in plain English. The model side has two conditions: a subword model, and a single-digit student trained with the cross-tokenizer distillation recipe from the non-BPE tokenizer paper. Both students are then given the same light multi-turn fine-tuning on Code-Feedback-style repair dialogues built from training templates only.

Every mismatch is labeled by its ratio signature: an exact power of ten, a match to a listed constant, a sign flip, a small rational or π factor, or an unstructured mismatch. We then report repair rates by error class. The claim predicts that the rendering arms help most on the structured classes and not at all on unstructured ones. That pattern would distinguish legibility from a general boost in effort.

Baselines

The naive reference point is one-shot Program-of-Thought on the same problems with no feedback, as in TheoremQA's protocol, scored on hidden instances. For context, SymPyBench reports about 65% exact match for Claude 3.7 Sonnet and Gemini 2.0 Flash in one-shot answering. That is a different protocol, with no code execution and no hidden-instance scoring, so it frames difficulty rather than serving as a target. No reported result was found on Lacuna for iterative execution-feedback repair on SymPyBench.

The strongest competitor that must be beaten is compute-matched resampling: best-of-n with the visible instance used as a filter, with n chosen so its total tokens equal those of the true-feedback loop. It is strongest because the loop may simply be resampling, and plateau results from VERINA and Yan et al. suggest a lot of the gain is exactly that.

The strongest existing refinement method is OpenCodeInterpreter-style multi-turn repair with raw execution feedback, using ICER's relaxed acceptance policy. On HumanEval, OpenCodeInterpreter-33B reports 79.3% single-turn and 82.9% with execution feedback (Pass@1, OpenCodeInterpreter). That 3.6-point gain is the kind of margin we expect raw feedback to give on numerical code as well. We run it with the same backbone sizes, round cap, sampling temperature, and token budget as every other arm, and tune prompts for all arms on a shared development split of templates.

The key ablation is the rendering axis, holding the model fixed, together with the tokenizer axis, holding rendering fixed. Beating the baselines means two things. First, true-feedback repair with structured rendering must exceed compute-matched resampling on variant-consistent accuracy by a margin outside seed variance, across at least three seeds and two model sizes. Second, the gain must concentrate in the structured error classes.

Why not just…?

Why not just resample more? This is the main threat. If compute-matched best-of-n equals the raw-feedback loop, current numeric feedback is contributing nothing. That is the first result this proposal would report, and it would motivate rather than sink the rendering arms. If best-of-n also equals the structured-rendering loop, the claim is false and we say so.

Why not just have the harness compute the ratio and say it in words? This is the cheapest fix and plausibly sufficient. We therefore treat it as a first-class arm, not a strawman. If it matches the digit-tokenized student, the recommendation becomes "harnesses should do diagnostic arithmetic." The tokenizer arm then serves only to test whether models can find signatures the harness designer did not anticipate, which we measure on the unstructured error class.

Why not just retrain with a digit tokenizer and skip the harness changes? The non-BPE tokenizer paper reports that single-digit chunking lengthens outputs 1.9 to 2.6 times and that dedicated number encodings degrade general benchmarks such as MMLU. A tokenizer change is therefore costly. It is justified only if it beats inference-time re-rendering on the same subword model, which the rendering arms test directly.

Why not just inject unit and constant tables into the prompt? TheoremQA found that adding definitions to the prompt gave no significant boost, which suggests the bottleneck is applying knowledge rather than having it. We include a constants-table-in-prompt arm without feedback so this can be checked on physics code specifically.

Decisive Checks

Three results would settle the question. The first is the shuffled-feedback control. If shuffled feedback performs as well as true feedback under raw rendering, the loop is not reading its numbers at all. The second is the error-class breakdown. Structured rendering should lift repair of power-of-ten and constant errors well above resampling, while leaving unstructured errors unchanged. If instead it lifts all classes equally, the effect is generic prompting. The third is the contamination split. We compare repair gains on canonical textbook parameter values against fresh ones. For the fine-tuned students, we also run a check following the code-contamination paper: a paired-leak probe with held-out templates. If gains appear only on canonical values, the loop is recalling answers, not diagnosing them.