Do Math Judges Misread Numbers? Tokenization-Controlled Tests of Step Verifiers

Automated judges now decide which solutions a model is rewarded for. These include reward models that score each step of a math solution, and large language models prompted to grade answers. A judge that cannot tell 48,317 from 48,371 will reward wrong arithmetic. There is concrete reason to suspect this happens. Most tokenizers, the components that cut text into the units a model reads, split numbers into arbitrary chunks such as three-digit groups. Studies of solver models trace a large share of arithmetic failures to these chunk boundaries. Nobody has checked whether the models that grade solutions inherit the same blind spot.

This note proposes to measure that directly and then test the cheapest fix. The measurement injects arithmetic errors into real solution traces, controlling exactly where each error falls relative to the judge's token boundaries. The fix exploits an asymmetry between judging and solving. A judge mostly reads numbers rather than writing them. Re-splitting the numbers in its input into single digits should therefore cost little. For solvers, the same change makes outputs two to three times longer.

Core Claim

The claim is that numeric errors missed by step-level judges are predictable from the judge's tokenization, and that the predictability is specific. Errors hidden inside a multi-digit token, and errors in operands whose length is not a multiple of the chunk size, should be missed far more often than equally large errors that change a whole token. The prediction carries over the failure anatomy that AI-rithmetic found in solvers. In that study, operand misalignment from three-digit token boundaries, together with carry failures, explains 87.9% of Claude Opus 4.1's addition errors and 62.9% of GPT-5's. Our version applies it to the act of checking someone else's arithmetic.

The second claim is about where the fix belongs. Are Non-BPE Tokenizers Ready to Improve LLM Numeracy? shows that moving a solver from three-digit to one-digit chunks raises pure-arithmetic accuracy from 27% to 83%. It also shows that the same change inflates generated length by 1.9–2.6×. That length cost is the main reason the paper finds no tokenizer that beats standard subword tokenization overall. A verifier pays most of that cost only on its input, which is processed in parallel. The trade-off that rules out digit-level tokenization for solvers may therefore not apply to judges. One caveat: judges that reason before scoring do generate text, and that text may restate numbers. We measure this rather than assume it away.

Research Program

The core artifact is a perturbation benchmark built on solution traces for GSM8K and MATH. The traces come from the policies used in Math-Shepherd, and the error injection follows the automated approach of FG-PRM. That approach reports over 94% injection success for its six error types. We restrict injection to its Calculation Error and Context Inconsistency types, the latter meaning a number copied incorrectly from earlier context. We add tokenization controls the original work does not have.

For each correct step, we generate erroneous variants matched on numeric magnitude of the error. The variants differ in three ways. The first is whether the corrupted digit sits inside a multi-digit token or forms its own token. The second is whether the operand length is a multiple of three. The third is whether the problem's numbers have been rescaled from two-digit to five- to eight-digit values. Rescaling preserves the logic of the solution while moving it into the length regime where Number Cookbook reports GPT-4o addition falling from near 100% to 15%. We also keep correct steps containing the same number formats, so that false rejections of sound work are measured alongside missed errors.

The primary quantity is a tokenization alignment gap for a judge $J$:

$$\Delta_J = \mathrm{Det}_J(\text{aligned}) - \mathrm{Det}_J(\text{misaligned}),$$

where $\mathrm{Det}_J$ is the rate at which $J$ flags injected erroneous steps. Here "aligned" means the corrupted digits coincide with whole tokens, and "misaligned" means they fall inside a token or straddle a boundary. Error magnitude is held fixed across the two conditions. The prediction is that $\Delta_J$ is substantially positive for judges whose tokenizers chunk digits. It should be near zero for judges whose tokenizers already split digits individually. We will verify each backbone's actual tokenizer behavior empirically before assigning it to a group, which gives a natural contrast without training anything.

The intervention arm tests three repairs of increasing cost. The cheapest is inference-time digit splitting. Numbers in the judged trace are re-encoded digit by digit before the frozen judge reads them, with no training. Adversarial Tokenization shows models largely keep meaning under non-canonical tokenizations, so this may work as is. The same work shows that alignment-type behaviors can degrade under such shifts, so judging quality may degrade too, and that is exactly what we test.

The middle repair trains a Math-Shepherd-style step reward model twice on identical Monte Carlo labels, once with standard tokenization and once with digit-split inputs. This isolates representation from data. The most expensive repair applies the cross-tokenizer distillation recipe from the non-BPE tokenizer study to the judge backbone. Including both training repairs addresses the fine-tuning paradox in Number Cookbook. That paper found that representation changes which help from-scratch models can hurt when grafted onto pretrained checkpoints, so we do not assume the cheap fix transfers.

Baselines

The naive reference point is an unmodified judge with standard tokenization, scored on the same traces. We use an off-the-shelf process reward model, a Math-Shepherd-style model on Mistral-7B, plus prompted instruction-tuned judges. On the original unrescaled data, the Math-Shepherd verifier reaches 89.1% GSM8K and 43.5% MATH accuracy when reranking 256 Mistral-7B samples, per Math-Shepherd. Reproducing these figures confirms our pipeline before we perturb anything. There is no reported result found on Lacuna for any judge's step-level detection rate under tokenization-controlled numeric perturbations, so those numbers come from this study.

The strongest learned-judge competitor is a reasoning model used as a judge, with the explicit evaluation plan of PlanJudge. Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases reports DeepSeek-R1 at 91.18% on RewardBench. It also reports that PlanJudge raises R1's BiasBench score from 65.00 to 97.50. This is the main rival because extended reasoning could re-derive each number and erase any tokenization effect. We run an open reasoning judge in this mode on the identical benchmark and report its generated-token count. Beating it means one of two things. Either a digit-split non-reasoning judge matches its detection rate at a fraction of the tokens, or digit splitting closes a residual alignment gap that the reasoning judge still shows.

The strongest overall competitor is a tool-grounded judge in the spirit of TaTToo. That system gives an 8B reward model Python and SQL calls and reports a 30.9% inference improvement for a 14B policy on tabular reasoning. We give the judge a Python interpreter, and separately we run a non-learned checker that extracts explicit equations with patterns and evaluates them. Both get the same traces and the same token budget for the judge's own text. Judges trained under the two tokenizations get the same labels, steps, and hyperparameter search budget.

The key ablation is the trained-reward-model pair that differs only in input tokenization. Any gain there is attributable to representation alone.

Why not just…?

Why not just give the judge a calculator? This is the real threat. For steps that state an explicit equation, we expect the pattern-plus-calculator checker to be near perfect and cheaper than anything learned. If it also wins on implicit steps, the honest conclusion is to use tools. The remaining case for a tokenization fix rests on steps where the arithmetic is implied rather than written, such as "so she has 4,317 left," or where a number was miscopied from earlier context. Turning such steps into checkable code requires a model to read the numbers correctly first, so the tool inherits the reading problem. AI-rithmetic documents copy errors even after correct code execution. Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics shows that generated code is often logically unsound. We split results by explicit versus implicit steps so the tool's domain of dominance is measured, not argued.

Why not just use a reasoning model as judge? Reasoning judges may recompute everything. However, AI-rithmetic finds that carry failures behave as independent per-carry errors, with success scaling as $(1-p)^n$, so longer computation alone should not remove them on long operands. Reasoning judges also carry length biases that only planning prompts suppressed in Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases. If the reasoning judge shows no alignment gap on the rescaled traces, the tokenization story fails for that judge class, and we will report it.

Why not just scale the judge? Tokenization Falling Short finds that structural token-level failures persist from 8B to 70B. That result concerns strings, not judging, so we measure $\Delta_J$ across at least two sizes per model family rather than lean on it.

Why not just retokenize the whole model to single digits? For generation, the 1.9–2.6× length cost is documented. For a judge, the question is whether input-only digit splitting suffices without retraining. That is the cheapest arm of the program, so this alternative is tested rather than dismissed.

Proposed Experiments and Decisive Checks

The first experiment estimates $\Delta_J$ with confidence intervals for each judge. Judges are grouped by verified tokenizer behavior, and the analysis is stratified by operand length modulo three and by rescaling factor. The decisive check is a sign test across the two groups. If chunking judges show a gap that single-digit judges do not, the error is tokenization-driven. If both groups show the same gap, the failures come from something else, such as attention to long numbers, and the fix arm becomes moot.

The second experiment applies the repairs and reports detection AUROC, false-rejection rate on correct steps, and total tokens processed and generated per judged trace. The third experiment checks whether any repair matters downstream. It reranks 256 samples on GSM8K, MATH, and their rescaled versions, following the Math-Shepherd protocol. A repair that closes $\Delta_J$ but leaves reranking accuracy unchanged would mean the blind spot is real but inconsequential at current solver error rates.

The proposal succeeds if three things hold together. Input-side digit splitting, or the matched trained reward model, must cut the gap substantially. It must match the reasoning judge's detection at far fewer generated tokens. It must beat the calculator-grounded judge on implicit and copy-error steps while conceding explicit ones. Any other pattern is a clear, publishable map of where each kind of judge should be trusted with numbers.