The Double Burden: Separating Language and Local Context in Multilingual Math Reasoning

Education researchers have long argued that children taught from imported textbooks carry a double burden. They must decode a foreign language and also decode foreign settings (unfamiliar names, markets, units, customs) before they can reach the concept being taught. The practical answer from that literature is localization: rewriting the material so the setting is familiar. Language models that are weak in low-resource languages are now being fixed with a machine version of the same move. Training data is regenerated around local personas instead of being translated from English. It works, but nobody knows why it works. The model could be gaining from better target-language text, from shedding translation artifacts, or from the local setting itself. Nor does anyone know whether the standard fix for weak languages, which is to reason in English first, quietly breaks exactly the problems that localization was meant to serve.

This note proposes a controlled test of the double-burden hypothesis for language models. We will build math word problems as matched minimal pairs. Each problem has one symbolically verified arithmetic skeleton and is rendered in English or a target African language, crossed with three settings. The first is an imported setting, meaning the Western framing inherited from translated GSM8K. The second is a surface-local setting, where only names and places change. The third is a load-bearing-local setting, where a local unit, currency subdivision, or market convention actually enters the computation, for example buying garri by the "paint bucket" or pricing in kobo. This design lets us measure a language penalty, a context penalty, and their interaction separately. The central prediction is that surface localization is nearly free, but load-bearing local context imposes a real penalty that compounds with the language penalty. We further predict that English-pivot methods repair the language penalty while enlarging the context penalty, because the pivot translation mishandles the local quantities. The same factorial design, applied to training data, tells us which kind of localization is worth paying for.

Core Claim

The closest prior work is Improving Multilingual Math Reasoning for African Languages. It generated 60,000 persona-grounded math problems in 9 African languages and showed they beat machine-translated BigMath and OpenMathInstruct for fine-tuning Llama 3.1 8B. Average AfriMGSM accuracy rose from 14.6% to 31.8%. That comparison changes several things at once. The problems differ in authoring (generated natively rather than translated), in cultural setting, and in the underlying problem distribution. It also leaves the synthetic answers unverified. Our claim is narrower and testable. Holding the arithmetic skeleton fixed, local context splits into two kinds. The surface kind costs models almost nothing. The load-bearing kind produces a context penalty that is super-additive with the language penalty in low-resource languages.

We will measure the penalties on accuracy $A(\ell, c)$ for language $\ell$ (English or target) and context $c$ (imported or local). The language penalty is $\Delta_L = A(\text{en},\text{imp}) - A(\text{tgt},\text{imp})$ and the context penalty is $\Delta_C = A(\text{en},\text{imp}) - A(\text{en},\text{loc})$. The interaction is

$$I = \big[A(\text{en},\text{imp}) - A(\text{tgt},\text{loc})\big] - \Delta_L - \Delta_C ,$$

where $I > 0$ means the two burdens compound rather than add. We expect $I \approx 0$ for surface-local items and $I > 0$ for load-bearing items.

This is a diagnostic claim with a training consequence. If the effect lives in the load-bearing items, then the name-swapping localization described in Position: LLMs Can be Good Tutors in English Education is cosmetic. That paper envisions LLMs lowering learners' cognitive load by swapping names, settings, and customs, but offers no empirical test. Worthwhile localization would then need the "deep reframing rather than literal translation" argued for in Revolutionizing Education through AI-Powered Inclusive Learning Systems.

Why This Is Open

Two lines of evidence motivate the claim, and they pull in opposite directions.

The first line shows that reasoning in English helps weak languages. Not All Languages Are Created Equal in LLMs introduced cross-lingual-thought (XLT) prompting, a template that has the model restate the problem in English, reason step by step in English, and answer in the target language. It lifted gpt-3.5-turbo on MGSM from 23.3 to 70.0. PLUG trains the model to write an English pivot before its target-language response. Language models are multilingual chain-of-thought reasoners found English reasoning often beats native reasoning at scale.

The second line shows that models know cultural facts but fail to use them. Culturally Aware NLI found GPT-4 above 90% on quizzes about cultural norms but with a 30% error rate when it had to apply a norm, even with the norm supplied in context.

Put together, a load-bearing local unit is exactly the kind of knowledge a pivot must carry across. It is also exactly the kind that models know without applying. None of these papers tests that collision. All of them evaluate on translated benchmarks such as MGSM, X-SVAMP, and AfriMGSM, whose problems keep an American grade-school setting.

Translationese Artifacts in Multilingual Language Model Evaluation adds a constraint on our design. Translated test items carry unnatural phrasing, which can make a model look weaker than it is. Models trained on translated data can also score well on translated tests and still fail on native text. For this reason, authoring (native-written versus machine-translated) must be controlled rather than left confounded with context.

Research Program

Benchmark. The first artifact is a minimal-pair benchmark in Swahili, Yoruba, Hausa, Nigerian Pidgin, and Zulu. These languages overlap AfriMGSM and AfriPersona-Instruct, so our results connect to existing numbers.

We start from roughly 300 arithmetic skeletons, each written as an executable program so the gold answer is computed rather than trusted. This directly addresses the unverified-answer weakness of AfriPersona. Each skeleton is rendered in four contexts. The first is imported. The second is surface-local. The third is load-bearing-local, where a local quantity convention enters the arithmetic. The fourth is load-bearing-local with a one-line definition of the convention given in the problem. That last condition is our version of the CALI setting where the norm is supplied and the model must still apply it.

Native speakers write each context in the target language directly from the English version. For a subset, we also produce machine-translated versions so that the translationese effect can be read off separately. Candidate local conventions are drawn from the African web sources behind AfriPersonaHub and then vetted by speakers. Every rendering is checked by running its program.

Evaluation. The second component runs the benchmark on off-the-shelf and adapted models, reporting $\Delta_L$, $\Delta_C$, and $I$ per language and context type. We will also log the English pivot trace under XLT and PLUG-style generation. We then score, against a gold mapping, whether each load-bearing entity survived the pivot with its quantitative meaning intact. A pivot that turns "paint bucket" into "bucket" and then assumes a size counts as a fidelity failure. This gives a direct measure of the "cultural erosion" that the pivot literature has never checked.

Training. The third component is a matched training study. At the 30,000-example budget used by AfriPersona, we fine-tune Llama 3.1 8B and Gemma-2-9B on data that shares skeletons and differs in one factor at a time. The four arms are machine-translated imported problems (the XCOT-style regime), native-written imported problems, native-written surface-local problems, and native-written load-bearing-local problems. We also run AfriPersona-Instruct itself at the same size.

Because AfriPersona reported that computing loss on prompt tokens helps in low-resource fine-tuning, we cross each arm with masked versus unmasked prompts. This checks whether any context effect survives that choice.

Baselines

The naive reference point is the untuned Llama 3.1 8B Instruct with direct answering. The file reports 14.6% average accuracy for it on AfriMGSM (Improving Multilingual Math Reasoning for African Languages). Every intervention must first clear this bar on our benchmark and on AfriMGSM.

The strongest data-side competitor is fine-tuning on AfriPersona-Instruct, which reached 31.8% average and 54.4% in Swahili on AfriMGSM with 30k samples. It is the strongest because it is the only existing method that localizes training data for these languages. We will run it with the same example count, epochs, learning-rate sweep, and prompt-masking setting as our arms. Our load-bearing-local arm beats it if it matches AfriPersona on AfriMGSM (within noise across three seeds) while clearly exceeding it on the load-bearing slice of our benchmark. If AfriPersona already closes the load-bearing gap, our hypothesis about which kind of context matters is wrong for training, and we will say so.

The strongest method-side competitor is XLT prompting from Not All Languages Are Created Equal in LLMs, which reported 70.0 average on MGSM for gpt-3.5-turbo against 23.3 for direct prompting. We will also run a PLUG-style pivot-tuned model trained on our imported arm. For XCOT-style alignment, the file reports 51.5% MGSM average at Llama-2-13B (XCOT). For LinguaLIFT, it reports 58.9% low-resource accuracy on MMWP (LinguaLIFT). We use these as external anchors rather than rerunning their full pipelines. Pivot methods are evaluated with identical decoding and token budgets. We also report tokens generated, since pivoting roughly doubles output length. The question is not whether pivots raise average accuracy (they will) but whether they raise it on load-bearing items as much as on imported ones. A load-bearing context penalty under pivoting that is larger than without pivoting would count as the confirming result.

The key ablation is the surface-local arm and test slice. It isolates the claimed active ingredient, load-bearing context, from mere cultural decoration.

Why not just…?

Why not just swap in local names, as the tutoring position paper suggests? This is the cheapest localization, and it may well be enough for the surface kind of context. Our benchmark includes it as its own condition. If surface-local items show no penalty and the surface-local training arm matches the load-bearing arm everywhere, then deep reframing is unnecessary for models. That would be a useful negative result. The CALI finding that knowledge fails at the application step (Culturally Aware NLI) is our reason to expect otherwise when the local entity carries a number.

Why not just use XLT or a PLUG pivot and skip localization? Pivoting is a real threat because its reported gains are large. Our pivot-fidelity scoring tests it directly. If English traces preserve load-bearing quantities and the pivoted context penalty is no worse than the direct one, then pivoting is sufficient and the curriculum argument weakens to a matter of cost and output language. We will report this outcome plainly if it occurs.

Why not just scale up AfriPersona-style generation? More persona data may close gaps simply through volume. However, AfriPersona conflates authoring, setting, and distribution, so scaling it cannot tell us which ingredient to scale. Our matched arms at equal size can. A scaling point at 60k for the best and worst arms will show whether the ranking persists with more data.

Why not just supply the definition of the local unit in context? If supplying the definition recovers imported-level accuracy, the penalty is missing knowledge, and retrieval would solve it. The definition-provided condition measures exactly this. A remaining gap would reproduce the CALI knowledge-application gap in a low-resource, multi-step setting.

Decisive Checks

The study succeeds as science whichever way the results fall, but four checks decide the claim.

First, the interaction term $I$ on load-bearing items must be positive and must exceed its value on surface-local items. This is tested with a bootstrap over skeletons in at least three of the five languages.

Second, the definition-provided condition must leave a residual penalty; otherwise the phenomenon is only missing knowledge.

Third, under XLT, the load-bearing penalty should grow relative to direct prompting, and pivot-fidelity failures should account for most of that growth.

Fourth, in training, the load-bearing-local arm should beat the translated-imported arm on the load-bearing slice by more than the surface-local arm does. This advantage must hold under both prompt-masking settings and must not cost accuracy on AfriMGSM or MGSM.

If all four hold, the education literature's double burden has a measurable analogue in language models, and localization budgets should go to load-bearing context rather than names. If the first fails, the models do not share the burden, and the finding is equally worth reporting.