Language models answer much worse when a prompt also contains irrelevant or misleading text. Adaptive distractors cut GPT-4o-mini's CommonsenseQA accuracy from 85.7% to 22.0% (Breaking Focus: Contextual Distraction Curse in Large Language Models). Separately, models built for one language often beat much larger multilingual models. Part of the reason is the tokenizer, the component that cuts text into the sub-word pieces the model actually reads. A multilingual tokenizer can split Thai into about 2.7 times as many pieces as a Thai-aware one; SeaLLMs cut Thai compression from 4.29 to 1.57 tokens (SeaLLMs -- Large Language Models for Southeast Asia). These two findings have never been connected. This note proposes that fragmentation is itself a source of distractibility. When words are chopped into short, generic pieces, irrelevant sentences share many more pieces with the question, even when they share no whole words with it. Those accidental overlaps pull the model's attention toward the distractors.
The claim is testable in a way natural-language studies cannot manage alone. We build a small artificial language in which the same sentences can be tokenized at different granularities. We inject distractors whose overlap with the question is controlled at the word level and at the piece level separately. We train small models from scratch on each version. We then carry the result to a real language by continuing pre-training of one base model on Thai text, with and without a Thai-aware vocabulary, under identical data and compute. If the hypothesis holds, part of the "monolingual advantage" is robustness to noisy context, and that has a mechanical cause that can be fixed. If it fails, the advantage lies elsewhere, and the length-and-overlap explanation can be retired.
Core Claim
At a fixed number of words, a model's accuracy loss from irrelevant context grows with the tokenizer's fragmentation ratio (sub-word pieces per word). The growth is mediated mainly by piece-level overlap between distractor text and the question. Sheer sequence length contributes less. Two consequences follow. First, a length-matched block of neutral filler should hurt far less than a fragmented distractor with the same token count. Second, holding fragmentation fixed while removing piece-level overlap from the distractors should remove most of the extra damage.
The measure throughout is retention: accuracy with distractors divided by clean accuracy. Using retention separates "easier to distract" from "simply weaker in this language."
Relation to Prior Work
Three papers motivate the gap by measuring distraction, but only in English and without tokenization as a variable. Breaking Focus finds distractors with a tree search and shows the failure is one of reasoning, not missing knowledge. Thinking in a Crowd shows that step-by-step reasoning makes misleading snippets more harmful on SciAux. Improving Zero-shot Reader by Reducing Distractions shows open-domain QA accuracy falling as retrieved documents grow from 5 to 100. We reuse their distractor types and their finding that telling the model to "ignore irrelevant information" does not help. What we add is language and tokenizer as controlled variables.
The closest multilingual work is Robust Contrastive Pretraining. It mines typos from Wikipedia edits in each language and trains multilingual encoders to resist them. Its main result is +18.4 noisy slot-labeling F1 on MultiATIS++. That work treats noise inside the sentence being classified. We treat whole irrelevant passages placed next to the relevant one, and we ask whether the tokenizer determines how damaging those passages are.
The artificial-language arm supplies the method. Endowing Neural Language Learners with Human-like Biases used miniature languages with word dropout to show that noise shapes how recurrent listeners comprehend. We adopt its logic of manipulating one pressure and watching the learned behavior. We move it to Transformers and to structured distractors rather than random dropout. Tamura et al. found that synthetic pre-training data only transfers when word frequencies follow natural power laws, so our toy lexicon is Zipfian.
The synthetic-error papers constrain the design. Synthetic Error Injection Fails to Elicit Self-Correction shows that randomly injected errors do not resemble the errors a model makes naturally: recognition fell from 94% on injected errors to 8% on the model's own errors. So we never evaluate only on our own injected distractors. We always also evaluate on distractors found adversarially against the model under test.
The language-specific pre-training papers supply the phenomenon, but not its cause. ArabicMMLU reports Jais-chat at 62.3% against near-chance multilingual open models. SeaLLMs report Thai GSM8K 62.0 against ChatGPT-3.5's 35.8.
Research Program
The artificial language. Entities, attributes and relations are drawn from a Zipfian lexicon of character strings built from a small alphabet. The alphabet is small on purpose: with few letters, pieces recur across unrelated words, as they do in fragmented scripts. Each example is a context holding one fact sentence plus zero to k distractor sentences, followed by a question about the fact. Because a grammar generates the corpus, word-level and piece-level overlap between each distractor and the question are known exactly.
Tokenizer granularities. We tokenize the identical corpus three ways: a whole-word vocabulary, byte-pair encoding (a standard method that builds a vocabulary by repeatedly merging frequent character pairs) at a medium vocabulary size, and a small vocabulary. These give fragmentation ratios of roughly 1, 3 and 8, spanning the Thai 1.57-versus-4.29 contrast and the larger ratios reported for Khmer.
Distractor conditions. Distractors are produced by controlled injection in four types. Neutral distractors are well-formed sentences with no overlap with the question. Word-overlap distractors share whole entity words with the question. Piece-overlap distractors share sub-word pieces but no whole words. Length-matched filler repeats one uninformative token.
Models and measurements. We train decoder-only Transformers of about 20M, 80M and 150M parameters from scratch, with three seeds per cell and the same number of training words, not tokens, in every cell. Besides retention, we measure the attention mass that answer-producing positions place on distractor tokens, aggregated per word so the three tokenizations are comparable. We also run causal tests: zeroing attention to piece-overlap positions should restore accuracy if the proposed mechanism is right.
The natural-language arm. We take one small open base model and continue pre-training it on the same Thai corpus twice: once with its original tokenizer and once with a vocabulary expanded SeaLLM-style. A fixed English share is mixed in, following SeaLLMs' warning that pure regional training erodes reasoning. Both runs get the same tokens-per-word-adjusted data and the same compute. We evaluate on Thai versions of SciAux-style questions (billhdzhao/SciAux), keeping helpful, irrelevant and misleading snippets and generating them natively in Thai rather than translating from English. We add Thai translated GSM8K with irrelevant passages inserted. As a secondary check with a second script, we evaluate on Bengali TyDiQA-GoldP from BenHalluEval with distractor passages inserted. Distractors come from two sources: our overlap-controlled injection, and adversarial distractors from the tree search in Breaking Focus, run separately against each model. Released Llama-2 and SeaLLM checkpoints serve as an observational sanity check only, because their training differs in more than the tokenizer.
Baselines
Clean accuracy. The do-nothing reference is each model's clean accuracy. Retention is defined against it, so a model that is merely stronger in Thai gains no credit.
The original-tokenizer model. The main comparison within the proposal is the model continued on Thai with its original, fragmenting tokenizer. It shares data, compute, learning-rate schedule and evaluation prompts with the expanded-vocabulary run, so any retention gap is attributable to tokenization. The expanded model "beats" it if its retention is higher by more than the seed-to-seed spread, measured with three continued-pre-training seeds at the smaller scale.
Distraction-robustness training. The strongest existing mitigation is preference fine-tuning (DPO, which trains the model to prefer one answer over another) on distracted-versus-clean question pairs, from Breaking Focus. We apply it to the original-tokenizer model with the same number of Thai preference pairs and the same tuning budget that the vocabulary change receives, tuned on a held-out split. Two published reference points exist, both in English. For the distraction problem, GPT-4o-mini falls from 85.7% to 22.0% on CommonsenseQA under adaptive distraction (Breaking Focus). For misleading context, Qwen3-8B falls from 96.22% clean to 77.03% with two misleading snippets and reasoning enabled (Thinking in a Crowd). No reported result was found on Lacuna for distraction robustness in Thai, Arabic or Bengali, so we establish those numbers ourselves.
Distractor-only fix. Distraction-aware answer selection from Improving Zero-shot Reader is run as a baseline that requires no retraining.
Ablation of the key ingredient. This is the piece-overlap-removed condition: same fragmentation, same length, but distractors re-sampled to share no pieces with the question. If fragmentation hurts only through length, this ablation changes nothing.
Why not just…?
Why not just blame sequence length? Fragmented text is longer, and attention might simply dilute over more tokens. This is the most serious rival, and it is a real threat. The length-matched filler condition settles it. If filler with the same token count damages retention as much as fragmented distractors, the overlap mechanism is wrong and we report a length effect instead. Under our hypothesis, filler should be far milder than piece-overlap distractors at equal token count.
Why not just say native models are better at the language? Clean-accuracy gaps like those in ArabicMMLU would make every native model look more robust in absolute terms. Retention normalizes this away. The controlled continued-pre-training pair also holds the language data fixed, so "better at Thai" cannot differ except through the tokenizer.
Why not just train robustness with DPO in English and let it transfer? This could plausibly work, and if it does, tokenizer changes matter less in practice. We test it directly. English-only distraction DPO is applied to the original-tokenizer model and evaluated on Thai. The self-correction results in Synthetic Error Injection Fails give reason to expect poor transfer when the training noise differs in distribution from the test noise. Here that difference would be English piece statistics versus Thai piece statistics. That is a prediction, not a result.
Why not just prompt the model to ignore irrelevant text? Breaking Focus reports that such instructions were ineffective. We include the instruction as a cheap control in the Thai arm, but do not expect it to matter.
Why not just expand the vocabulary, since SeaLLMs already did? If vocabulary expansion is the whole answer, this project still explains why it helps. It also predicts where it will not help: distractors that share whole words with the question. That tells practitioners whether they need tokenizer work, robustness training, or both.
Decisive Checks
Four outcomes, each with a stated consequence, settle the claim.
In the artificial language, retention should fall monotonically with fragmentation ratio for piece-overlap distractors. It should stay roughly flat for neutral and filler distractors. Removing piece overlap should recover most of the gap. If retention falls with fragmentation equally for filler, the mechanism is length, not overlap.
In the attention analysis, per-word attention mass on piece-overlap distractors should rise with fragmentation. Ablating that attention should restore accuracy. If attention rises but ablation restores nothing, the attention shift is incidental rather than causal.
In Thai, the expanded-vocabulary model should retain more accuracy than the original-tokenizer model on both injected and adversarially searched distractors. The gap should be comparable to or larger than what matched-budget distraction DPO achieves. If the gap appears only on our injected distractors, it reflects our generator, not the model.
The Bengali replication checks that the effect is not specific to Thai script.
References
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- SeaLLMs -- Large Language Models for Southeast Asia
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Robustification of Multilingual Language Models to Real-world Noise with Robust Contrastive Pretraining
- Endowing Neural Language Learners with Human-like Biases: A Case Study on Dependency Length Minimization
- Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation?
- Synthetic Error Injection Fails to Elicit Self-Correction In Language Models
- ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
- BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali