Small language models pre-trained mostly on one language or region, such as SeaLLM for Southeast Asian languages or Jais for Arabic, often beat much larger multilingual models on local tasks. They are weaker at multi-step reasoning, though, because step-by-step worked solutions barely exist in those languages. The obvious fix is to translate English worked solutions or have a teacher write them in the target language, then fine-tune on the ones that reach the right answer. Translated and synthetic rationales are noisy in a particular way. Many reach the right answer while doing no work: the model has already settled on an answer internally, and the written steps are decoration. A correctness filter cannot see this.
We propose to read that hidden "settling point" directly. Following Drop the Act, we train a small probe on the regional model's hidden states that predicts, at each step of a rationale, whether the model has already committed to its final answer. Because the probe's target is an answer such as a number, not a word, its labels can be generated automatically in any language. We use it twice. First, as a diagnostic: for each language, we split the reasoning gap into cases where the model never reaches the right answer internally and cases where it reaches it but fails to say it. Second, as a data filter: we keep only synthetic native-language rationales in which the student's internal commitment arrives when the rationale actually computes the answer, and follows the same trajectory as the English version of the problem.
Core Claim
The claim has two parts. Among correct synthetic rationales in low-resource languages, a large share are internally performative, and fine-tuning on them teaches answer pattern-matching rather than reasoning. Filtering by the student's own commitment trajectory should therefore transfer reasoning into a regional model better than correctness filtering does, at equal data volume, without damaging the native-language ability that gives regional models their edge.
The diagnostic part also makes the claim falsifiable. If a language's gap turns out to be mostly a failure to verbalize an answer the model already holds internally, commitment filtering should not help there, and we will say so.
What Exists and What Does Not
Several studies motivate the proposal by locating the cross-lingual gap inside the network. Disentangling Language Understanding and Reasoning Structures finds language-specific neurons concentrated in Llama 3.1's reasoning layers; deactivating them costs up to 27.4% on Japanese MGSM. Language-Specific Latent Process Hinders Cross-Lingual Performance shows that larger models form isolated per-language subspaces that steering toward English cannot fix. Enhancing Low-Resource Language Reasoning argues low-resource prompts "under-elicit" reasoning features and recovers 25–33% of the gap with test-time sparse-autoencoder steering at one layer. All three study general multilingual models at inference time. None of them builds training data, and none examines regional models.
These findings also constrain the design. Since the native-language circuits carry reasoning, we avoid any method that forces every language through an English pivot. XCOT does exactly that with translated GSM8K and cross-lingual distillation, and it reaches 51.5% MGSM average with Llama-2-13B. Why Does Zero-Shot Cross-Lingual Generation Fail? shows that excessive cross-language invariance causes accidental English output.
Two methods are reused directly. The first is ProFIL's forced-answering probe from Drop the Act. That work is English-only and uses the probe to zero out reinforcement-learning rewards, whereas we use it to curate multilingual supervised data. The second is the observation from Geometry of Decision Making that commitment shows up as a peak and then a fall in intrinsic dimension (the number of directions the hidden states actually vary along) across layers. We use this as an independent check that the probe tracks a real event, but that work only studied English multiple-choice final tokens.
On the regional-model side, SeaLLMs and ArabicMMLU document the regional advantage behaviorally but never look inside the models. Is continuous CoT better suited for multilingual reasoning? is the closest training-time alternative: it compresses reasoning into latent tokens on a 1B model and does not analyze why that helps.
Research Program
Matched model pairs. The most important asset is a controlled comparison. The SeaLLM line, as described in SeaLLMs, extends Llama-2's vocabulary and continues pre-training on regional text, cutting the Thai token ratio from 4.29 to 1.57. We compare a SeaLLM checkpoint against its Llama-2 parent, so regional pre-training is the main difference. Jais-chat, which scores 62.3% on ArabicMMLU, is compared against a similarly sized multilingual model. Gemma-2-9B and Qwen2.5-7B serve as multilingual reference points because the steering paper used them.
Probe construction. For each model and language, we sample rationales on GSM8K-style problems in that language. We cut each rationale after step $k$ and force an answer. The label is whether this forced answer equals the eventual final answer. An attention probe on frozen hidden states learns to predict that label. The commitment step $c$ is the first step at which probe confidence exceeds a threshold $\tau$. We tune $\tau$ on English only and check that the dip in intrinsic dimension falls near $c$ in each language. No human annotation is needed at any point.
Gap decomposition. On MGSM, every native-language failure is assigned to one of three categories. Elicitation failure means the model never commits to the gold answer internally. Verbalization failure means it commits to the gold answer but emits something else. Theater means it commits before any arithmetic has appeared in the rationale. Comparing SeaLLM with Llama-2 tells us which category regional pre-training shrinks and which it leaves alone. This partly answers the open question about where the regional advantage lives: in tokenization and verbalization, or in reasoning itself.
Data synthesis and filtering. For each problem we build candidate native rationales from three sources: machine translation of English GSM8K solutions, the regional model's own samples, and a multilingual teacher's samples. Correctness filtering keeps those with the right final answer. Commitment filtering additionally requires two things. The student's commitment step $c$ must fall at or after the step where the rationale first states the answer's computation. The student's sequence of probe-decoded intermediate answers on the native rationale must agree with its sequence on the English version of the same problem. Because this agreement is measured in answer space, it never asks native hidden states to look English. We then fine-tune the regional model on the filtered set, matching example count across conditions.
Baselines
The naive reference is the regional model with no added training, prompted with native chain-of-thought. The strongest reported number for this reference is SeaLLM-7B-v2.5 at 72.3 on Vietnamese and 62.0 on Thai translated GSM8K, against 55.0 and 35.8 for ChatGPT-3.5 (SeaLLMs). We re-run it under our prompts and do not copy these numbers across setups.
The strongest training competitor is correctness-filtered native rationale fine-tuning: the same candidate pool, keeping every rationale with the right answer. It isolates our key ingredient. It uses the same base model, the same number of training examples (we subsample the correctness set down to the size of the commitment set), and the same learning-rate sweep and steps. Beating it means a statistically significant MGSM gain averaged over target languages, measured with three seeds.
We also run XCOT on the regional model as the English-pivot alternative; it reports 51.5% MGSM average on Llama-2-13B. We run CODI from Is continuous CoT better suited for multilingual reasoning? as the latent-reasoning alternative; it reports 12.28% versus 6.84% for text chain-of-thought fine-tuning on zero-shot Urdu with a 1B model. Both get the same data budget.
At test time we compare against sparse-autoencoder steering from Enhancing Low-Resource Language Reasoning, which reports 25–33% gap recovery and Korean MATH500 rising from 49.2% to 52.4% on Gemma-2-9B. We report gap recovery in the same terms so the two are comparable.
Two further ablations separate the parts of our filter: commitment-timing alone without cross-lingual trajectory agreement, and a random filter matched to the commitment filter's keep rate.
Why not just…?
Why not just filter rationales by correct final answer? This is the real threat, and the main experiment is built to settle it. In English, Drop the Act found correct but performative chains common enough that removing them during reinforcement learning added 5.2 points on GSM8K. We expect translated chains to be worse. If the size-matched correctness baseline ties us on MGSM and MATH500, the probe adds nothing for data curation, and we will report that result.
Why not just translate the question to English and reason in English? Cross-lingual Prompting found Translate-En below interactive alignment. ArabicMMLU shows GPT-4 falling from about 80% on translated MMLU to 72.5% on native exams, which means culturally grounded questions do not survive translation. We run translate-test on MGSM, where it is likely competitive, and on ArabicMMLU, where we expect it to lose.
Why not just steer at test time? Steering is cheap, but it recovers at most a third of the gap and degrades in large models with separated subspaces (Language-Specific Latent Process). It also has to be re-applied on every query. We also test whether the two combine.
Why not just use a bigger multilingual model? The scale results in Language-Specific Latent Process and Statement-Tuning cut against this: a 560M encoder beats Llama 3.1 70B by 10.5 points on XStoryCloze. We still include a 70B multilingual model as a ceiling.
Proposed Experiments and Decisive Checks
The main evaluation uses MGSM in Thai, Bengali, Swahili, Telugu, Japanese and Chinese, plus Vietnamese and Thai translated GSM8K and MATH500 in Korean and Vietnamese. Training data excludes every target language for the zero-shot condition. In a second condition, translated data in that language is included.
Retention of native ability is checked on ArabicMMLU for Jais and on Sea-bench for SeaLLM. For language-specific neurons identified by the LAPE method (a way of finding neurons that fire for only one language) from Disentangling, we measure how much their activation profiles change before and after training. The prediction is that commitment filtering preserves these neurons better than XCOT.
Four checks decide the proposal. First, the probe must transfer: an English-tuned threshold should mark commitment within one step of the intrinsic-dimension dip in each language, or the whole signal is suspect. Second, on the size-matched data, commitment filtering must beat correctness filtering; otherwise the central claim fails. Third, the gains must concentrate in languages that the decomposition labels elicitation-limited, and must be absent where the gap is verbalization. This cross-check would show the mechanism, not just the score. Fourth, in the SeaLLM and Llama-2 pair, the decomposition should show which failure type regional pre-training shrinks. That gives the first internal account of the monolingual advantage, whichever way the result falls.
References
- Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning
- Disentangling Language Understanding and Reasoning Structures in Cross-lingual Chain-of-Thought Prompting
- Language-Specific Latent Process Hinders Cross-Lingual Performance
- Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer
- XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning
- Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution
- Geometry of Decision Making in Language Models
- SeaLLMs -- Large Language Models for Southeast Asia
- ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
- Is continuous CoT better suited for multilingual reasoning?
- Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
- Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models