Does the Grader Pay the Token Tax? Auditing LLM Exam Scoring for Tokenization Bias

Schools and testing bodies increasingly rent commercial large language models to grade student answers. This is a cost-cutting move of the kind the commercialization-of-education literature describes. These models do not read words. They read subword tokens, fragments produced by a compression algorithm that splits rare or morphologically complex words into many small pieces. Prior work has shown that languages whose words shatter into many tokens get worse model accuracy and higher bills. That evidence is correlational across languages, though, and it concerns the model answering exam questions. Nobody has tested the question that matters for assessment: when the same correct answer is written two equally valid ways, and one way happens to break into more tokens, does the automated grader give it a lower score?

We propose a controlled audit that answers this question. The audit builds matched pairs of student responses that mean the same thing and are graded identically by native-speaker examiners, but that differ in how many tokens they produce. It then measures how much the grade and the per-response cost move as token count moves. Because the audit needs only black-box access to a grading service, a public exam board can run it on a vendor it cannot inspect. That turns a hidden technical bias into a number that can be written into a procurement contract.

Core Claim

The claim is that tokenization harms students in automated assessment through a channel that has not been measured: grading sensitivity to how a response is spelled and built from word parts, independent of its content. This channel should appear within a language, among answers of equal quality. It cannot be explained away by the usual confounders of cross-language comparisons, such as less training data, translation artifacts, or cultural mismatch in the questions. If the effect is real, a student who writes a correct answer using a longer agglutinated form, a legitimate suffix that triggers byte-level fallback, or a decomposed Unicode encoding produced by their keyboard will be marked down more often. The same student will also cost the institution more to grade. The proposal predicts a monotone relationship between the token-count gap within a pair and the probability that the grade flips from pass to fail. It also predicts that this relationship survives length normalization for likelihood-based graders.

Where This Sits

The closest work is The Token Tax: Systematic Bias in Multilingual Tokenization. It regresses accuracy of 10 models on AfriMMLU against token fertility, meaning the average number of tokens per word. It finds that fertility explains up to 50% of accuracy variance and that each extra token per word costs up to 18 percentage points. It frames the cost side in terms of training and inference compute. We depart from it in three ways. The model's role is grader, not test-taker. The manipulation is within-language and content-matched, not a cross-language regression in which fertility is tangled with resource level. And the cost unit is the institution's per-response bill under per-token pricing. IrokoBench and Belebele supply the parallel exam content we build on. Both treat exams purely as capability probes, which motivates asking what happens when the model sits on the other side of the exam.

Three mechanism papers tell us which variations to construct. Gambardella et al. show that a valid Japanese suffix can trigger byte fallback and raise perplexity enough to make models prefer ungrammatical text. That is exactly the failure a likelihood-based grader would inherit. Truong et al. show that tokenizers break negative affixes, which matters for answers whose correctness hinges on negation. Silva Plata shows that standard tokenization fuses Aymara evidentiality and other suffixes into opaque blocks, and supplies a 17,856-pair Spanish–Aymara corpus we reuse for the Aymara arm.

Two papers constrain the design. Evaluating Morphological Alignment of Tokenizers in 70 Languages finds that whether token boundaries match morpheme boundaries explains almost none of the downstream variance (recall $R^2 = 0.024$). We therefore do not build the study around morphological alignment. We manipulate token count, the compression variable that paper found to dominate. The survey by Shani et al. argues that cross-lingual gaps shrink once sequence length and encoding are normalized. This is a real warning: our effect might vanish under simple normalization. We therefore test normalization as a baseline rather than assuming it fails.

On the commercialization side, Dong and Yang use principal-agent theory to show how private partners in public vocational training pursue goals that diverge from the institution's mandate under information asymmetry. Li and Paramalingam document how marketized schooling shifts costs and priorities away from equitable access. We borrow the principal-agent lens only for one concrete purpose: to define an audit statistic that a principal (the exam board) can observe without seeing the agent's (the vendor's) tokenizer. We make no causal claims about educational markets. Those papers are correlational and descriptive, and our study does not change that.

Research Program

The first component is a matched-response dataset. We start from AfriMGSM word problems and AfriMMLU items drawn from IrokoBench, add Japanese items translated from the same English sources, and add short Aymara answer prompts built from the Silva Plata corpus. For each item, native-speaker annotators write correct and incorrect free-text answers, each with a short justification, so the grader has to judge reasoning and not just an option letter.

Each response then receives variants from three sources, ordered from most mechanical to most linguistic. The first is Unicode normalization: the same text in composed (NFC) and decomposed (NFD) forms, which real students produce depending on device and keyboard. The second is attested alternative spellings and word-segmentation conventions, such as clitics written attached versus separate, where the language's orthography permits both. The third is native-speaker paraphrases that swap an analytic construction for a synthetic, morphologically dense one, or add a legitimate suffix such as an evidential or honorific. Two independent examiners must certify that each pair deserves the same grade. Pairs where they disagree are discarded. Every variant is token-counted under each grader's tokenizer where that tokenizer is public, and through billed-token counts reported by the API where it is not.

The second component is the grader panel. It includes proprietary chat models used as rubric graders through their public APIs. It also includes open models with different tokenizers (Llama 2, Llama 3, Gemma), used both as rubric graders and as likelihood scorers that grade an answer by how probable the model finds it. Using several open tokenizers on the same text gives natural variation in token count across tokenizers, alongside the variation we create within a tokenizer.

The third component is the audit statistic. For a matched pair of responses $(x, x')$, let $g(\cdot) \in \{0,1\}$ be the pass/fail grade and $n(\cdot)$ the token count under the grader's tokenizer. We fit

$$ \Pr[g(x') \neq g(x)] = \sigma\big(\alpha + \beta \,\log(n(x')/n(x)) + \gamma^\top z\big), $$

where $\sigma$ is the logistic function and $z$ holds item and language fixed effects. The coefficient $\beta$ is the tokenization sensitivity of the grader. Under the null hypothesis that the grader responds only to content, $\beta = 0$. We report $\beta$ alongside a cost ratio, the billed tokens for $x'$ divided by those for $x$. Together these two numbers form a vendor-auditable profile.

Baselines

The naive reference point is the ungraded status quo: a grader that sees raw student text with no preprocessing. Our effect sizes are measured against it. It is also the configuration most commercial deployments would actually ship.

The strongest competitor is translate-then-grade. Each response is translated into English and graded there, the assessment analogue of translate-test. IrokoBench reports that translate-test raised Gemma 2 27B by +17.6% and LLaMa 3.1 70B by +21.0% when answering IrokoBench tasks. It is the best-evidenced fix in the file. We will run it with the same graders, the same rubric prompts, and one fixed translation system chosen in advance, and we will not tune it separately. The proposal beats it only if translation either leaves a nonzero $\beta$ or introduces a new failure. That failure would be a higher false-fail rate on the paraphrase-arm pairs whose correctness depends on a morpheme that translation drops, such as an Aymara evidential. No reported result found on Lacuna for any grader, translated or not, on in-language free-text exam responses. For reference, the file's best reported answer-side figure is GPT-4o's 59.0% IrokoBench average.

The second competitor targets likelihood graders: length-normalized scoring, in which log-likelihood is divided by the number of tokens, or alternatively by the number of characters. This is the normalization Shani et al. suggest removes much of the apparent gap. It is run on identical pairs with no extra tuning. If it drives $\beta$ to zero, the likelihood-grader part of our claim is refuted.

The key ablation separates the three variant sources. If $\beta$ is large only for the NFC/NFD arm, the problem is a preprocessing bug that input normalization fixes, not a tokenization-morphology problem. The claim requires a nonzero $\beta$ on the paraphrase arm after normalization. A second ablation uses ByT5, a model that reads raw bytes instead of subword tokens, fine-tuned as a grader on the same training split. It tests whether any sensitivity is specific to subword vocabularies.

Why not just…?

Why not just compute per-language fertility and price, as the Token Tax paper does? That analysis already exists in The Token Tax. It cannot separate tokenization from everything else that differs between English and Amharic. A board could reasonably attribute a cross-language gap to question translation or model knowledge. The matched-pair design holds language, item, and content fixed, so any nonzero $\beta$ is attributable to token count. That is what makes it usable as a contractual audit and not only as a descriptive statistic.

Why not just translate responses to English before grading? This is the most serious threat, and it may well remove most of the within-pair sensitivity for rubric graders. We treat it as the main baseline, not a strawman. The experiment settles it on the paraphrase arm. If translate-then-grade gives zero $\beta$ and no rise in false fails on morpheme-dependent answers, we will report that translation is an adequate mitigation and that the remaining contribution is the cost audit. The evidence against assuming this outcome is Silva Plata's demonstration that evidential and relational suffixes do not survive translation cleanly. Translation also adds a second billed service per response.

Why not just normalize Unicode and divide scores by length? Input normalization will almost certainly eliminate the NFC/NFD arm. That is why we report that arm separately and do not rest the claim on it. Length normalization is a real fix for likelihood graders and is tested head-on. It does not apply to rubric-prompted chat graders, which output a grade and not a likelihood. The Gambardella byte-fallback result suggests the damage there is a jump in model confidence, not a smooth length effect.

Why not just retrain the tokenizer with Parity-aware BPE? Parity-aware BPE cuts the cross-language compression Gini coefficient from 0.064 to 0.011, which is a genuine improvement. However, exam boards rent graders; they do not pretrain them. Parity-aware BPE also equalizes compression across languages, not across alternative forms within one. It is a remedy for vendors to adopt, and our audit is how a board would check whether they did.

Decisive Checks

The main test is whether $\beta$ on the paraphrase arm, after Unicode normalization, is significantly positive for at least the likelihood graders and one rubric grader in two or more languages. Examiner-certified equivalence is the ground truth. The second check is whether $\beta$ tracks tokenizer choice among the open models on identical text: Llama 2's smaller vocabulary should yield larger token-count gaps and more flips than Llama 3 if the mechanism is right. The third check is whether translate-then-grade and length normalization close the effect. If both close it completely, the proposal reduces to a cost audit, and we will say so. Throughout, we report the cost ratio beside $\beta$, so that a board can see whether the students most often marked down are also the most expensive to grade.