When a language model answers a numerical question by writing a short program and running it, the usual way to make it reliable is to sample many programs and take a majority vote over their outputs. This fails in a specific way: several wrong programs can print the same wrong number. We propose to execute every candidate program not only on the original inputs but also on a family of rescaled versions of them, for example every dollar amount multiplied by 2, 10 or 1000. We then accept a candidate only if its output responds to rescaling in the lawful way a correct solution must, meaning it is multiplied by a fixed power of the scale factor. Candidates are grouped by their behaviour across that whole family of inputs, and the vote is taken over groups. Candidates that break the scaling law get the violation back as a concrete error message and are asked to repair it.
The idea comes from a small literature on neural networks without bias terms. Removing the constant offset $b$ from every layer and using ReLU activations forces a network to be positively homogeneous, $f(\alpha x)=\alpha f(x)$. Such networks generalize to input scales far outside training where ordinary networks collapse. The constant offset is the part that memorizes the training scale. LLM-written programs have the same pathology in symbolic form: a hardcoded constant copied from the example, an INF = 10**9 sentinel, a magic threshold. Each one works on the input at hand and silently breaks when magnitudes change. Our claim is that homogeneity, which that literature enforces architecturally, can be checked cheaply at test time on arbitrary generated code. Used this way, it adds an agreement signal to voting and an error signal to iterative repair.
Core Claim
Self-consistency over program outputs, as in PoT-SC from Program of Thoughts Prompting, compares candidates at a single input point. Two programs that agree there may still compute different functions. We call the set of inputs obtained by rescaling the same-unit quantities of a problem by a set of factors its scale orbit. Comparing candidates across the scale orbit separates programs that agree only by coincidence. It also detects a class of error, constants that should have been variables, that no amount of resampling at the original point can reveal.
Concretely, the model writes a program that binds input quantities to named variables and tags each with a unit group, such as money, count or length. We rescale one unit group at a time by factors $\alpha \in \{2, 10, 10^3\}$ and execute. For candidate $p$ with base output $y_0$ and rescaled outputs $y_\alpha$, the homogeneity residual is
$$ r(p) = \min_{k \in \{-2,\dots,2\}} \max_{\alpha} \left| \log|y_\alpha| - \log|y_0| - k\log\alpha \right|, $$
where $k$ is the integer degree the program appears to have in that unit group. A dollar total has $k=1$, a ratio of dollar amounts has $k=0$, and an area in length units has $k=2$. A correct solution to a dimensionally consistent problem should have $r(p)\approx 0$. The vote is then taken over candidates whose output vectors along the whole orbit agree, weighted down by their residuals. Candidates with large residuals, and candidates that crash, are sent back with the failing rescaled input and the observed-versus-expected ratio before the final vote.
The new ingredient is not "vote after repairing," which is a natural extension of Distilling LLM Agent into Small Models with Retrieval and Code Tools. That work discards crashing candidates and votes over the rest. The new ingredient is a verifier derived from a symmetry of the problem. It needs no reference tests and costs only interpreter calls.
Research Program
The work has three parts.
Unit tagging. We extend the PoT prompt so that the model declares a unit group for each bound input variable. Tagging is itself stochastic. Candidates therefore carry their own tags, and we measure how often a tagging error, rather than a program error, produces a large residual. Where tags disagree across candidates, we use the majority tagging.
Orbit construction and scoring. We execute each candidate on the base input plus three rescalings per unit group, compute $r(p)$, and cluster candidates by their rounded orbit output vectors. This extends voting from a discrete scalar to a short vector. The file identifies voting over structured outputs as a gap in self-consistency; this stays tractable because the vectors are low-dimensional.
Repair. Candidates that crash or exceed a residual threshold receive a feedback message of the form "when all prices were multiplied by 10, your output changed by a factor of 1.0; a total cost should scale by 10." We allow at most two repair rounds. That limit follows the report in CodeTransOcean that compiler-feedback debugging plateaus after two to three rounds.
A known risk constrains the repair design. SlopCodeBench documents models satisfying local feedback by deleting edge-case handling. A repaired program could likewise become homogeneous by dropping a legitimate constant, such as a flat fee that genuinely does not scale. We guard against this in two ways. First, the base-input output of a repaired candidate must still land in some existing cluster or be supported by independent samples. Second, the unit tags let a declared "fixed fee" variable stay unscaled.
We also acknowledge where the idea does not apply. Floors, ceilings, integer division and piecewise rules break homogeneity even in correct programs. A problem such as "buses hold 40 people; how many buses" is not homogeneous in people. We therefore compute the residual only for unit groups whose rescaling keeps the program's discrete operations exact: integer factors for counts, and money only when it is not divided into counts. We report separately how often the method correctly abstains.
Baselines
The naive reference point is greedy decoding: a single Program-of-Thoughts sample executed once. It shows how much of any gain comes from sampling at all.
The strongest existing method on this question is PoT-SC from Program of Thoughts Prompting. It samples many programs, executes each once, and takes a majority vote over outputs. That paper reports 80.0% accuracy on GSM8K with Codex, and on financial QA it reports a 15–20% gain over chain-of-thought rather than per-dataset absolute figures. Codex is not the model we will use, so we will rerun PoT-SC ourselves on the same open-weight model, prompts, datasets and sampling temperature. We give it the same total generated-token budget as our method. Because our repair rounds consume tokens, PoT-SC gets correspondingly more initial samples. Beating it means higher accuracy at matched tokens on GSM8K, GSM-Hard and FinQA, with the largest margin expected on GSM-Hard, whose inflated numbers are exactly where hardcoded constants break.
The second strong competitor is SAG from Distilling LLM Agent into Small Models with Retrieval and Code Tools, which prunes crashing candidates before voting. No reported per-dataset result is on Lacuna for SAG; the file gives only model-size comparisons. We run it on the same small Qwen2.5 students it was designed for, at matched budget.
Two ablations isolate the key ingredient. The first is "SAG plus crash repair": crashes get error-trace feedback, but there is no orbit, no residual and no orbit voting. If our gains come merely from salvaging crashes, this ablation will match us. The second is "orbit voting without repair," which scores and clusters but never re-prompts. It separates the value of the verifier from the value of the feedback.
For the graph-code experiments, the numerical reference is the learned bias-free model of Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous GNNs. It reports a relative loss of about 0.03 on weighted shortest path with edge weights scaled up to 24× the training range, and 100% success on 5,000-node graphs after training on graphs with fewer than 34 nodes. LLM-written programs that reach comparable robustness at those scales, without training, count as matching it.
Why not just…?
Why not just draw more samples for plain self-consistency? This is the most serious threat. It is why every comparison is at matched generated tokens: orbit executions add interpreter time but no tokens, so the budget comparison is fair. The argument that it falls short is structural rather than empirical. More samples at the same input point cannot separate two programs that agree there. The file notes that self-consistency collapses onto shared misconceptions, and a hardcoded constant copied from the problem text is exactly such a shared error. The experiment that settles this is the recovery rate on questions where plain majority vote is wrong. If scaling the PoT-SC sample count closes that gap, the orbit adds nothing and we will say so.
Why not just let the model write its own unit tests and debug against them? Model-written tests with model-written expected outputs inherit the model's misconceptions. If it believes the answer is 42, its test asserts 42. The scale check asks the model only for unit tags, which are easier to get right than answers. It then derives expectations mechanically from the observed base output. We include a model-written-tests baseline at matched budget. If it matches orbit voting on GSM-Hard, the symmetry is not buying anything beyond generic self-testing.
Why not just tell the model in the prompt to avoid hardcoded constants and reason about units? This costs nothing and may capture part of the gain. We include it as a prompt-only baseline and also add the instruction to our own method. The evidence that instructions alone are not enough is indirect. CodeTransOcean reports that verbalized reasoning before code can hurt global correctness, and that execution feedback, not instruction, drove its debugging gains (48.57% to 52.57% DSR over three rounds for GPT-3.5-turbo). If the instruction alone recovers most of the improvement, the contribution shrinks to a prompting trick, and we would report that.
Why not just build a bias-free neural verifier instead of checking programs? A homogeneous network could score candidates. However, it would have to be trained per task family, and the file notes that existing scale-equivariant operator networks such as Graph Neural Preconditioners are often trained per instance. Executing the candidate program on rescaled inputs checks the property exactly, with nothing learned. The architecture literature supplies the diagnosis and the invariant, not the component.
Proposed Experiments
Math word problems. The main study runs on GSM8K, SVAMP, AQuA, GSM-Hard, FinQA and TATQA with one open-weight model at two sizes, reporting accuracy at matched token budgets. The financial sets matter because their quantities span millions to billions, and PoT's gains there show that computation offloading already helps. The open question is whether scale checks help further.
Graph code. A second study turns the synthetic graph tasks of Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous GNNs into a code-synthesis benchmark. The model writes shortest-path and component-counting functions from small example graphs from the Erdős–Rényi, KNN, planar and lobster generators. These are tested on graphs with edge weights scaled to 24× and sizes up to 5,000 nodes. Shortest-path length is degree-1 homogeneous in the weights, and component count is degree 0, so the residual is exact here and sentinel-constant bugs are common.
Structured code. A small third study applies the residual to CodeTransOcean's LLMTrans set on numerical functions whose source-language behaviour defines the expected scaling. The goal is to test whether the signal adds to DSR@K beyond compiler feedback.
Decisive checks. Four results decide the proposal.
The first is recovery on hard cases: among questions where PoT-SC's majority is wrong but some sampled candidate is correct, what fraction does orbit voting flip to correct? This is the direct test of the claim that the orbit breaks coincidental agreement.
The second is residual calibration: across all candidates, how well does $r(p)$ separate correct from incorrect programs, measured by area under the ROC curve, on problems where the method does not abstain?
The third is a negative control on problems dominated by floors, ceilings and fixed fees. The method must abstain or remain neutral there. A drop in accuracy on these problems would show the repair loop is eroding legitimate constants in the way SlopCodeBench warns about.
The fourth is the matched-token ablation ladder: greedy, PoT-SC, SAG, SAG plus crash repair, orbit voting without repair, and the full method. If the gain disappears at the step that adds the orbit, the idea is wrong.
References
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
- Distilling LLM Agent into Small Models with Retrieval and Code Tools
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
- Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous GNNs
- Graph Neural Preconditioners for Iterative Solutions of Sparse Linear Systems