When a large language model reasons step by step in a language it has seen little of, the text it writes often looks wrong before the answer is. It repeats itself, slips into English mid-sentence, produces words with implausible length or letter patterns, or writes sentences of erratic length. This proposal tests a simple claim. The surface statistics of a model's own native-language reasoning trace, measured against the surface statistics of ordinary human text in that language, predict whether the trace's answer is correct. These are the kinds of counts that readability research has long used, such as sentence length, word length, repetition, character patterns, and script mixing. The signal needs no labels, no translated step-level annotations, and no second model. If it holds, it can decide when an expensive fix is worth paying for. The fixes in question are translating the problem into English and reasoning there, or sampling reasoning paths in many languages and voting.
The opportunity matters because the strongest current remedies for the multilingual reasoning gap are costly and applied indiscriminately. English-pivot prompting and multi-language self-consistency raise accuracy substantially, but they spend extra tokens on every query, including the many that native reasoning already solves. Meanwhile, the only learned step-level verifiers for non-English reasoning exist for a handful of languages, because they require translated supervision. A cheap verifier built from surface counts would fill exactly the slot where low-resource languages lack tooling. It must also be shown to generalize to languages it was not calibrated on. Showing it works only within the 250-problem test sets it was tuned on would not be enough.
Core Claim
We define a surface drift score for a reasoning trace. It is the distance between the trace's profile of surface features and the profile of reference human text in the same language, after math symbols and numerals are masked. We claim three things. First, drift predicts trace correctness on multilingual math reasoning better than trace length alone, and at least as well as the model's own token-level confidence. Second, this holds when the drift-to-correctness mapping is fit on some languages and tested on held-out languages and a held-out task. Third, a router that runs one native chain-of-thought and escalates to costlier strategies only when drift is high can recover most of the accuracy of always-on multi-language ensembling at a fraction of the tokens.
The novelty lies in where readability measurement is applied. It is not used to judge a finished text for human readers, and it is not used as a training reward. It is used to judge whether a model is operating inside its competence in a given language.
Grounding in Prior Work
Three papers motivate the gap. Language models are multilingual chain-of-thought reasoners introduced the Multilingual Grade School Math benchmark (MGSM), with 250 problems human-translated into ten languages. It showed that reasoning in English often beats reasoning in the native language, which tells us native traces fail often. Disentangling Language Understanding and Reasoning Structures in Cross-lingual Chain-of-Thought Prompting found that Llama 3.1 uses language-specific neurons whose deactivation degrades reasoning in that language only, by up to 27.4% in Japanese. It also found lower-resource languages more fragile. We read this as evidence that failures in a weak language have language-specific causes, which is plausibly visible in that language's surface output. That paper probes internals; we look only at emitted text. Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought showed that deliberate English–Korean mixing can help. This constrains our design: script mixing cannot be treated as automatically bad. The drift score therefore separates mixing confined to logical scaffolding and numerals from uncontrolled drift into English prose, and we test whether the distinction matters.
The escalation targets come from the cross-lingual prompting line. Not All Languages Are Created Equal in LLMs proposed cross-lingual-thought prompting (XLT), a template that restates the request in English, reasons in English, and formats the answer back. Cross-lingual Prompting proposed sentence-by-sentence alignment into English (CLP) plus voting over reasoning paths in several languages (CLSP). AutoCAP lets the model choose 7–10 languages and weight their paths by its own judgment. All three apply their full cost to every query and use no external signal of when native reasoning has already succeeded. We reuse them unchanged as the expensive arms of a router.
The feature design comes from readability research. BATS showed that parameterized surface features beat dense OpenAI embeddings at predicting human simplicity judgments, and that they are interpretable. Work on Filipino and Central Philippine languages (Diverse Linguistic Features for Assessing Reading Difficulty of Educational Filipino Texts; BasahaCorpus) showed that word and sentence counts and syllable-pattern densities carry most of the signal in low-resource languages. It also showed that English formulas do not transfer. That work measures static human text against grade levels. We measure model text against a language's own norm. This also explains why we do not use Flesch-style formulas, which ARTS found align poorly with human judgments even in English.
The closest verifier is Demystifying Multilingual Chain-of-Thought in Process Reward Modeling. It trains step-level reward models on translated PRM800K and Math-Shepherd in six non-English languages. Its coverage stops where translated supervision stops; for example, it omits Telugu, Bengali, and Thai from its training data. Our score needs only unlabeled native text.
Research Program
The drift score has three parts. The feature extractor computes language-agnostic surface statistics per trace. These are sentence-length mean and variance, word-length distribution, type–token ratio, n-gram repetition rate, the fraction of characters outside the language's main script (excluding masked numerals and math), and the per-character perplexity under a small character n-gram model of the language. That last feature serves as a portable stand-in for the syllable-pattern features that worked in Filipino.
The reference profile for each language comes from two free sources. One is the human-written MGSM problem statement itself, which gives a per-query reference. The other is FLORES sentences, which give a per-language reference. The score is a standardized distance between trace and reference profiles. We test both an unweighted version and a version where a small logistic model weights features. The weighting model is fit with leave-one-language-out protocols only.
The router runs one greedy native chain-of-thought. It accepts the answer if drift falls below a threshold. Otherwise it escalates, either to XLT or to a multi-language ensemble. We trace out the full accuracy-versus-tokens curve by sweeping the threshold. As a secondary use, we replace AutoCAP's self-assigned path weights with drift-based weights.
Models are open-weight so that the confidence baselines are computable: Llama 3.1 at 8B and 70B, and Qwen2.5-7B. Benchmarks are MGSM across all its languages, with Swahili, Telugu, Bengali, and Thai as the low-resource core, and MSVAMP as a held-out task. XNLI and PAWS-X serve as a stress test of whether the signal survives outside arithmetic, where traces are short.
Generalization is treated as a first-class question because MGSM has only 250 problems per language. Fitting and evaluating on the same pool would reproduce the cross-validation illusion: optimistic numbers that do not travel. All reported numbers use one of two protocols. The first is leave-one-language-out, in which the score is fit on nine languages and tested on the tenth. The second is leave-one-task-out, in which the score is fit on MGSM and tested on MSVAMP. Every figure carries bootstrap confidence intervals over problems.
Baselines
The naive reference point is a single native chain-of-thought with no verification or escalation. It sets the floor on both accuracy and cost.
The always-pivot reference is XLT run on every query. The file reports zero-shot MGSM rising from 23.3 to 70.0 on gpt-3.5-turbo with XLT (Not All Languages Are Created Equal in LLMs). We rerun it on our models, since that figure is for a different system.
The strongest existing method is AutoCAP, which AutoCAP reports at 78.6% average MGSM accuracy, against 75.5% for CLSP. The file does not state the backbone model for those figures, so we reimplement both CLSP and AutoCAP on our backbones with their published prompts. AutoCAP is the right target because it is the most accurate multilingual prompting method in the file and also the most expensive. It therefore defines the accuracy ceiling that a cheap router should approach.
For fairness, every method uses the same backbone, decoding temperature, and prompt language. The router's threshold is tuned only on held-out languages, under the same protocol as the score. We report accuracy at matched average token budgets. The router counts as beating AutoCAP if it is within its confidence interval of AutoCAP's accuracy at no more than half of AutoCAP's average tokens, or if it is strictly more accurate at equal tokens.
The verifier comparison uses PRM-MULTI from Demystifying Multilingual Chain-of-Thought in Process Reward Modeling. That paper reports it beating outcome reward models by over 6 points on MGSM with Llama-3.1. We compare on the languages it covers. On Telugu, Bengali, and Thai we report it as untrained-on, which is the regime our method targets.
Three ablations isolate the key ingredient. Trace length alone serves as the drift score. Script-mixing fraction alone is a trivial "did it fall back to English" detector. Mean token log-probability is the model's own confidence. The drift score must beat all three in correctness-prediction AUROC on held-out languages for the core claim to stand. A final ablation removes the reference profile and uses raw features without comparison to native text. It tests whether the "relative to the language's norm" part does any work.
Why not just…?
Why not just always translate into English and reason there? Always-pivot is strong and is one of our arms. However, Cross-lingual Prompting found plain translate-then-solve loses nuance relative to interactive alignment. Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought documents topic drift and dropped local entities under pure English pivoting. Pivoting also pays its cost on queries native reasoning already solves. The accuracy-versus-tokens curve settles this directly. If always-pivot dominates the router at every budget, the proposal fails.
Why not just use trace length, or a fixed token budget? This is a real threat. Length is itself a surface feature, and failed traces are often long. Correct, Concise and Complete shows that fixed budgets break hard problems, which argues against truncation but not against length as a verifier. We therefore include length-only as an ablation with the same fitting protocol. If it matches drift on held-out languages, we will report that the readability features add nothing beyond length.
Why not just use the model's own confidence? Token log-probability is free on open models and is the most natural competitor. One reason to expect drift to add information is that low-resource generation can be confidently fluent in a degenerate way, as with repetitive loops, which confidence rewards. That is a hypothesis, not evidence. The head-to-head AUROC comparison, and a combined model, decide whether drift carries complementary signal.
Why not just train a multilingual process reward model, or fine-tune the reasoner, as XCOT does? XCOT and PRM-MULTI both work where translated supervision exists, and where it does they may well win. Our comparison against PRM-MULTI on its covered languages measures that gap honestly. The case for drift rests on the uncovered languages, and on fitting nothing per language beyond a reference profile.
Decisive Checks
The first decisive experiment is correctness prediction. We measure AUROC of drift versus length, script mixing, and log-probability for native MGSM traces, under leave-one-language-out, with the four low-resource languages reported separately. Drift must clear all three single-feature baselines outside their confidence intervals on held-out languages.
The second is transfer. A score fit on MGSM must keep most of its AUROC on MSVAMP. A collapse there would mean we learned MGSM's quirks rather than a property of degraded generation.
The third is the cost curve. The router, escalating to AutoCAP, must reach within noise of AutoCAP's accuracy at half its tokens or less.
A useful negative outcome is equally informative. If drift predicts correctness only in high-resource languages, where traces are already clean, the hypothesis that surface degradation tracks competence is wrong in exactly the regime that matters. We would report it as such.
References
- Language models are multilingual chain-of-thought reasoners
- Disentangling Language Understanding and Reasoning Structures in Cross-lingual Chain-of-Thought Prompting
- Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
- Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
- AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought
- BATS: BenchmArking Text Simplicity 🦇
- Diverse Linguistic Features for Assessing Reading Difficulty of Educational Filipino Texts
- BasahaCorpus: An Expanded Linguistic Resource for Readability Assessment in Central Philippine Languages
- ARTS: Assessing Readability & Text Simplicity
- Demystifying Multilingual Chain-of-Thought in Process Reward Modeling
- Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning
- XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning