Step-Level Language Choice for Multilingual Medical Reasoning

A multilingual language model asked a medical question in Chinese or Japanese has two kinds of knowledge to draw on. Universal biomedical facts, such as how a drug class works, were mostly learned from English text. Local facts, such as a regional drug brand, a national treatment guideline, or a Traditional Chinese Medicine concept, may exist only in the native language. Prior work forces one choice for the whole answer: reason entirely in English, reason entirely in the native language, or let a teacher model mix the two however it likes. We propose deciding the language one reasoning step at a time. The model breaks a question into sub-questions. For each sub-question in the training data, we measure whether answering that step in English or in the native language makes the final answer more likely to be correct. These measured preferences are then distilled into a compact model that writes its whole reasoning chain in one pass, choosing a language for each step.

The opportunity is concrete. Reasoning in English is reliably better on math (Language models are multilingual chain-of-thought reasoners). Yet models also hold knowledge they express better in a native language than in English (Language Specific Knowledge: Do Models Know Better in X than in English?). Medical licensing exams mix both kinds of content inside a single question. Whole-question routing cannot resolve that conflict. Step-level routing based on measured outcomes might, and testing that is cheap.

Core Claim

In multilingual medical question answering, the best language for an intermediate reasoning step depends on what the step asks, not only on what language the question is in. Language labels for each step can be derived from counterfactual rollouts, meaning we hold the rest of the chain fixed and swap one step's language. A compact model trained on chains labeled this way should beat three alternatives under equal data and compute: whole-chain English reasoning, whole-chain native reasoning, and teacher-heuristic language mixing.

What is new is the supervision signal. Every language switch in the training traces is justified by an observed change in downstream correctness. In prior mixed-language work, the switches are whatever a prompted teacher chose to produce.

Relation to Prior Work

The closest work is Language-Mixed CoT (Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought). It prompts Qwen3-32B to write Korean–English mixed reasoning traces and distills them into 4B–35B students. The students gain an average of 18.6 points on Korean benchmarks, and the paper shows that pure English pivoting loses local entities. That paper motivates our proposal and is also its main threat. Its switching policy is a prompt instruction ("use English for logic, Korean for terms"), the traces are produced in one undivided pass, and the evaluation covers general Korean benchmarks rather than medicine. We keep the idea of mixing but replace the heuristic with step-level outcome measurement, and we test in medicine, where the split between universal and local knowledge is sharpest.

Two papers supply evidence that native-language computation is real and not just noise to be translated away. The neuron-ablation study in Disentangling Language Understanding and Reasoning Structures in Cross-lingual Chain-of-Thought Prompting finds language-specific reasoning neurons in Llama-3.1 whose removal cuts Japanese MGSM accuracy by up to 27.4%. LSKExtractor (Language Specific Knowledge) chooses a language per query and gains up to 10.4% relative on CultureAtlas. That second paper also defines our coarser baseline: one language per question, rather than one per step.

Pivot methods constrain the design. PLUG (PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning) trains a fixed sequence: interpret the query, think in English, answer in English, then answer in the target language. With 2,000 such examples it beats 96,000 monolingual ones. XCOT (XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning) aligns English and low-resource reasoning distributions on grade-school math. Self-translate (Do Multilingual Language Models Think Better in English?) shows that the model's own translation unlocks English reasoning. All three make English the default for every step. Our method recovers that behavior as a special case whenever the measured labels say English always wins.

From modular decomposition we take two techniques. The first is splitting questions into sub-questions (Unsupervised Question Decomposition for Question Answering). The second is collapsing a multi-pass pipeline into one decoding pass with explicit state labels, which the radiology "building blocks" model does at 1 pass instead of 2.5 with comparable accuracy (Building blocks for complex tasks). The single-pass collapse answers the latency objection to pipelines. It also separates us from MedCoT (MedCoT: Medical Chain of Thought via Hierarchical Expert), which is English-only and runs a multi-model pipeline on Gemini Pro.

Finally, the perturbation audit in Right Diagnoses, Decorative Reasoning warns that medical reasoning chains can be decorative, meaning they do not actually drive the answer. If so, step language would not matter at all. We treat that possibility as a hypothesis to test, and it is the first check below.

Research Program

The base model is MMed-Llama 3 8B (Towards building multilingual language model for medicine). It already reads all six MMedBench languages and medical text, so any gain we measure comes from how it reasons, not from new domain knowledge.

Building the training labels has four stages, all using the same model, so no proprietary orchestrator is involved. First, the model decomposes each MMedBench training question into a short sequence of sub-questions, written in the question's language. Second, for every sub-question it produces one answer in English and one in the native language. Third, we score each step by swapping only that step's language and measuring the effect:

$$\Delta_i = \Pr(\text{correct} \mid s_i^{\text{EN}}, s_{-i}) - \Pr(\text{correct} \mid s_i^{\text{native}}, s_{-i})$$

Here $s_i$ is step $i$'s answer, $s_{-i}$ is the other steps held at their default language, and each probability is estimated from $N=8$ sampled completions of the final answer. Fourth, steps with $\Delta_i$ above a margin are labeled English, steps below the negative margin are labeled native, and ties keep the native language so local wording is preserved.

The labeled chains are formatted as single-pass state-tagged sequences, of the form "sub-question, language tag, step answer, … final answer." The model is then fine-tuned on them. At inference, the model chooses each step's language itself, in a single decoding pass.

The analysis asks what the learned policy actually does. CMExam's expert annotations on five dimensions, including "Medical Fundamentals" (Benchmarking Large Language Models on CMExam), let us test a falsifiable prediction: English labels should concentrate on fundamentals steps, and native labels on steps involving local drugs, guidelines, and Traditional Chinese Medicine.

Baselines

The naive reference point is MMed-Llama 3 answering with native-language chain of thought, as released. That paper reports 67.75% average accuracy on MMedBench, against 62.79% for base Llama 3. This is the strongest reported number for a compact model on our main benchmark. Every trained variant below starts from this checkpoint.

The English-anchoring baseline uses the same model, translates the question itself (self-translate), and reasons entirely in English, following Do Multilingual Language Models Think Better in English?. It is also fine-tuned on English-only chains built from the same decomposed training questions. That training equalizes data with our method, as PLUG-style pivot tuning would.

The strongest competitor is Language-Mixed CoT. We reproduce the recipe of Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought on MMedBench by having a teacher write mixed traces under the paper's switching instruction, then distilling them into MMed-Llama 3. There is no reported result on Lacuna for this method on medical benchmarks; its reported figures are 64.0 average across nine Korean benchmarks for the 35B model and +18.6 points for compact models. To keep the comparison fair, we run it twice. One version uses Qwen3-32B as the teacher, as in the original. The other uses MMed-Llama 3 itself as the teacher, matching the compute and data available to our method.

The coarse-routing baseline chooses one language per question, LSKExtractor-style. Its oracle labels come from the same rollout budget, so it isolates the value of per-step choice over per-question choice.

Two ablations isolate the key ingredient, which is outcome-measured labels. The random-routing ablation assigns step languages at random while matching the English fraction our policy learns. The heuristic-routing ablation keeps steps native whenever they contain a named entity detected by a tagger, and uses English otherwise.

All trained variants use the same questions, the same number of fine-tuning steps, and the same hyperparameter search budget. We also report generated tokens per question. As external references, GPT-4 zero-shot scores 61.6% on CMExam, against a 71.6% human average.

Our method counts as beating the field only if it exceeds the strongest trained baseline by more than the seed-to-seed spread, with 3 seeds and paired bootstrap tests on MMedBench average accuracy, and does so without raising token count by more than about 20% over English-only reasoning.

Why not just…?

Why not just distill Language-Mixed CoT, which already mixes languages? This is the real threat. A prompted teacher may already place switches roughly where our measurements would. The matched-teacher comparison settles the question. We also directly measure how often the heuristic teacher's switches agree with the measured $\Delta_i$ labels. If agreement is high and accuracy ties, the honest conclusion is that heuristic mixing is enough for medicine, and the contribution shrinks to the diagnostic analysis.

Why not just reason in English throughout, since English chains of thought keep winning? On math they do: PaLM-540B reached 55% on MGSM with English reasoning, beating native reasoning (Language models are multilingual chain-of-thought reasoners). Medical exams, however, contain local content that translation strips out, the topic-drift failure documented for Korean. The CMExam breakdown by dimension will show whether English loses specifically on non-fundamentals questions. If it does not, the step-level labels will say "English" nearly everywhere and our method reduces to English anchoring.

Why not just pick one language per question? Per-question routing is simpler and already reported to help (Language Specific Knowledge). It cannot help a question that needs English pharmacology and a native guideline at once. The oracle per-question baseline gives that approach its best possible chance, so if it ties our method, step granularity was unnecessary.

Why not just bridge an English medical specialist with a multilingual encoder, as LangBridge does? LangBridge raised MetaMath-13B on MGSM from 40.5% to 53.3% with one linear layer (LangBridge: Multilingual Reasoning Without Multilingual Supervision). By construction, though, it funnels everything into the English model, and the paper itself reports losses on high-resource languages. It therefore cannot use the native-language knowledge we target. We treat it as complementary and do not run it as a main baseline.

Proposed Experiments

The first decisive check is that reasoning steps actually matter. We use the perturbation audit, corrupting a step and measuring whether the answer changes. If $|\Delta_i|$ is near zero for most steps on MMedBench, the chains are decorative, step routing has nothing to work with, and we stop.

The main experiment compares all baselines on MMedBench across its six languages. We test on CMExam both after Chinese training and zero-shot, guarding against the narrow-benchmark overfitting seen when Huatuo scored 12.9% on CMExam. For each method we report accuracy by language and by CMExam dimension.

MGSM serves as a control where local knowledge is nearly absent. There, the learned policy should route almost every step to English and match English-only reasoning. If it routes many math steps to the native language, the labels are noise.

The random-routing and heuristic-routing ablations test whether the outcome-measured labels, rather than mixing as such, drive any gain. The single-pass model is compared against the multi-pass labeling pipeline to confirm that collapsing into one pass keeps the accuracy, as it did for radiology event extraction.

Finally, we run an inspection by language. Using the neuron-attribution method of Disentangling Language Understanding and Reasoning Structures, we check whether native-routed steps activate that language's reasoning neurons more strongly. This links the behavioral policy to the mechanistic finding.