Large language models now grade other models' outputs. They pick preferred answers for training, flag unsafe content, and check math solutions for reinforcement learning. These judges never see text directly. They see it after a tokenizer has cut it into subword pieces, and the same character string can be cut in many valid ways. Attacks on safety models show the cut matters. Inserting emojis inside words, or simply choosing a non-standard split of unchanged text, can flip a guardrail's verdict. Nobody has asked the basic measurement question for judges in general: how often does a judge's verdict on a preference, reward, or correctness question change when the text stays fixed and only its segmentation changes? Nor has anyone asked which known judge failures that sensitivity actually explains.
This note proposes to define segmentation invariance (the verdict should depend only on the characters, not on how they were split into tokens) as a measurable property of judges. We would measure it across standard judge benchmarks and enforce it by training: the judge learns to agree with itself across alternative segmentations of the same string. A truly segmentation-free byte-level judge serves as a reference point for perfect invariance. We do not offer it as the solution, because current byte-level models are far too weak to reason like a judge. The deliverable is a diagnosis that separates segmentation-driven judge errors from semantic ones, plus a cheap training fix aimed only at the former.
Core Claim
We claim that a measurable share of judge errors comes from segmentation rather than meaning. We expect this share to be concentrated in two places. The first is character-level perturbation attacks. The second is non-English text in heavily fragmented scripts. We further claim that a consistency objective across segmentations removes this share at little cost to accuracy on clean inputs. We also make a falsifiable negative prediction: the "master key" reward hacks of One Token to Fool LLM-as-a-Judge are mostly not segmentation-driven, so invariance training should leave them largely untouched. If that prediction fails, the finding is itself new, since that paper treated master keys purely as a data problem.
The motivating evidence comes from attack papers that stopped short of judges in general. Emoji Attack breaks subword boundaries with emojis and lowers safety judges' unsafe-prediction rate by 12% on average (ShieldLM drops from 71.9% to 3.5% detection). Its authors call for "token-invariant" judges but build none. Adversarial Tokenization changes zero characters, picks only non-canonical token paths, and bypasses LlamaGuard and ShieldGemma at rates exceeding 80%. It argues that alignment is trained on one canonical segmentation only and recommends tokenization-agnostic training. It does not test generative judges, reward models, or pairwise ranking. Tokenization Falling Short supplies the mechanism: a word's embedding and the sum of its subword embeddings have cosine similarity of only 0.21. On the multilingual side, M-RewardBench reports accuracy drops as high as 13% for reward models outside English, and finds that better translation recovers only 1–3%. That leaves room for a representational cause, which our per-language measurements can test.
Research Program
The first stage is measurement. For a judge input $x$, let $S(x)$ be the set of valid tokenizations of its character string. A tokenization here means any sequence of vocabulary items that concatenates back to exactly $x$. We sample from $S(x)$ in two ways. Random samples use BPE-dropout, which randomly skips merge steps of the tokenizer. Worst-case samples use the decision-diagram search of Adversarial Tokenization, retargeted to maximize the judge's verdict change rather than safety bypass. We report two quantities for each benchmark item: the random flip rate and the worst-case flip rate.
We then intersect these quantities with the judge's errors on canonical input. This yields an estimate of how many errors are "segmentation-explained", meaning some nearby segmentation yields the correct verdict and the canonical one does not. We run the analysis on JudgeBench, RewardBench, and M-RewardBench per language, where we correlate flip rate with tokens-per-byte. We also run it on the master-key inputs and on RobustJudge's attack suite from LLMs Cannot Reliably Judge (Yet?).
The second stage is intervention. We fine-tune open judges, including a Qwen2.5-7B generative judge (the same family Master-RM builds on), with this objective:
$$\mathcal{L} = \mathrm{CE}\big(y,\, p_\theta(\cdot \mid s_0(x))\big) + \lambda\, \mathbb{E}_{s \sim D(x)}\, \mathrm{KL}\big(p_\theta(\cdot \mid s_0(x)) \,\|\, p_\theta(\cdot \mid s)\big)$$
The symbols are defined as follows. $y$ is the gold preference or correctness label. $s_0(x)$ is the canonical tokenization. $p_\theta(\cdot\mid s)$ is the judge's verdict distribution when it reads segmentation $s$. $D(x)$ is a mixture of BPE-dropout samples and worst-case paths found by the search. $\lambda$ weights the consistency term. The idea borrows from Cross-Tokenizer LLM Distillation through a Byte-Level Interface, which treats raw bytes as the common ground between incompatible vocabularies. We apply the same logic within one vocabulary. Every segmentation of the same bytes should produce the same verdict. Our objective works at the level of the verdict rather than at byte-level next-token prediction. That paper reports that byte-level prediction objectives hurt instruction following on IFEval, and a judge must follow rubric instructions.
The third stage is a byte-native reference. We fine-tune a token-free model in the style of MambaByte, which reads raw bytes with a linear-time state space model and resists character corruptions, as a pairwise preference classifier on the same data. It is invariant by construction. The question is how much accuracy that invariance costs at sub-1B scale. MambaByte was evaluated only up to 972M parameters and only on language modeling, so we expect weak judging accuracy. This arm calibrates the trade-off. It is not meant to win.
Baselines
The naive reference point is the unmodified subword judge on canonical input. For the strongest existing systems we quote reported numbers. On JudgeBench, GPT-4o reaches 50.86% with a vanilla prompt and 56.57% with the Arena-Hard prompt, and Claude-3.5-Sonnet reaches 64.29% (JudgeBench). On M-RewardBench, GPT-4o averages 81.1% (M-RewardBench). On RewardBench, ArmoRM-Llama3-8B scores 89.0 (RewardBench). We found no reported result on Lacuna for verdict flip rate under segmentation change for any judge, so the first stage establishes those numbers itself. Proprietary models enter only the measurement stage, through APIs that accept text, using character-level perturbations such as emoji insertion as a proxy. Exact non-canonical token paths cannot be submitted to them.
The strongest defense competitor is input re-tokenization. RobustJudge reports that it lowers average attack success rate to 16.67% (LLMs Cannot Reliably Judge (Yet?)). We apply it to the same base judge, with the same inputs and attack budget. Beating it requires a lower attack success rate together with no loss of clean accuracy, since RobustJudge notes that re-tokenization degrades benign performance.
The strongest training competitor is adversarial augmentation in the style of Master-RM. Its authors report near-zero false positive rates on lead-in master keys (One Token to Fool LLM-as-a-Judge). We reproduce it on the same base model, with the same number of training examples and the same tuning budget.
Two ablations isolate the key ingredient. The first trains with BPE-dropout but without the consistency term. It asks whether exposure to alternative segmentations alone suffices, which is the mitigation Tokenization Falling Short found only partial. The second uses the consistency term with random segmentations only, without worst-case paths.
Two conditions define beating the baselines. The consistency-trained judge must cut worst-case flip rate by at least half relative to the dropout-only ablation. It must also stay within one point of the base judge's clean accuracy on JudgeBench and RewardBench.
Why not just…?
Why not just re-tokenize inputs canonically before judging? Re-tokenization does help on RobustJudge. But Adversarial Tokenization shows modern tokenizers are not bijective, so canonicalizing a string does not undo every attack. Emoji Attack notes that attackers can switch to other delimiters that a filter does not anticipate. Re-tokenization also does nothing for the multilingual case, where the canonical segmentation is itself the fragmented one. The head-to-head comparison on RobustJudge and Emoji Attack settles which defense holds up, and the M-RewardBench arm tests the case re-tokenization cannot reach.
Why not just augment training with adversarial examples, as Master-RM does? For the specific shortcuts seen during training, this works very well, and we expect it to beat us on master keys. Its authors acknowledge it leaves mid-process keys open. JudgeBench shows that judges fine-tuned on narrow preference data (PandaLM, JudgeLM) can fall below random on hard reasoning. Our objective covers the entire segmentation set of every training string rather than a list of attack strings. The test is generalization: we train on emoji and random-split perturbations, then evaluate on held-out worst-case token paths and unseen delimiters.
Why not just average the judge over several segmentations at inference time? This is the most serious threat. Averaging over $k$ sampled segmentations approximates the invariant verdict directly and needs no training. We run it at $k = 4, 8, 16$ as an upper reference. Our method amortizes that ensemble into a single forward pass. If a cheap $k=4$ ensemble matches the trained judge on worst-case flip rate, the training contribution reduces to a cost saving, and we will say so. The ensemble remains exposed to worst-case search that targets the average, which we also test.
Why not just use a reasoning model? JudgeBench reports that o3-mini reaches 80.86%. Yet Emoji Attack found o3-mini still vulnerable, and chain-of-thought increased false positives on master keys. Longer reasoning does not repair what the model reads. We include a chain-of-thought version of the base judge to measure whether reasoning lowers flip rate.
Proposed Experiments
The decisive check is the error decomposition. If fewer than a few percent of canonical-input errors on JudgeBench and RewardBench are segmentation-explained, invariance is a security property only and not an accuracy lever, and the note's scope narrows accordingly.
The second check is the multilingual correlation. Per-language flip rate on M-RewardBench should track tokens-per-byte and the accuracy drop from English. If the consistency-trained judge recovers more than the 1–3% that better translation gave, that is evidence for a representational component of the multilingual tax.
The third check is the master-key negative control. We test segmentation variants of ":" and "Thought process:". If false positive rates do not change, master keys are semantic shortcuts. If they collapse, the vulnerability has a tokenization cause that the original work missed.
Finally, the byte-native reference charts accuracy against invariance across scale. It tells us whether waiting for token-free judges is a realistic alternative or a distant one.
References
- One Token to Fool LLM-as-a-Judge
- Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
- Adversarial Tokenization
- Tokenization Falling Short: On Subword Robustness in Large Language Models
- M-RewardBench: Evaluating Reward Models in Multilingual Settings
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges
- RewardBench: Evaluating Reward Models for Language Modeling
- LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
- Cross-Tokenizer LLM Distillation through a Byte-Level Interface
- MambaByte: Token-free Selective State Space Model