Language-Bridged Soft-Risk Training for Multilingual Reasoning

When a language model is trained with reinforcement learning on math problems in many languages, its worst failures cluster in the languages furthest from its dominant training data, such as Telugu, Swahili and Thai. These are also the languages where training signal is weakest. If every sampled answer to a Telugu problem is wrong, the learning signal for that problem is exactly zero. Risk-averse methods, which try to fix the worst cases, run into the same wall. They focus training on the lower tail of outcomes, and when that tail is uniformly zero, focusing on it yields nothing. We propose to break this deadlock with a shortcut that multilingual data provides for free. Parallel benchmarks contain the same problem in many languages, so the language of a problem can be treated as a dial the trainer controls. For a problem that is failing completely in the target language, we temporarily present a partly translated version, with some sentences rendered in a closely related, better-served "anchor" language. We then turn that mixing fraction down as the model starts succeeding, and tighten the risk-averse objective only once the tail carries signal.

Put in two sentences: dynamic risk adaptation methods ease a policy into risk-aversion, but they either follow a fixed clock or need a simulator whose difficulty they can control. In multilingual reasoning, the mix between a distant target language and a related anchor language is such a controllable difficulty knob, so we can close the loop per language and per problem, and test directly whether linguistic proximity or plain English makes the better bridge.

Core Claim

Risk-averse reinforcement learning, meaning training that optimizes the worst outcomes rather than the average, fails in a specific way. Efficient Risk-Averse Reinforcement Learning (CeSoR) proves that when the lower tail of returns is flat, conditional value-at-risk (CVaR) policy gradients vanish. CVaR is the average return over the worst α-fraction of outcomes. The paper calls this failure "blindness to success." CeSoR fixes it with two tools. The first is a cross-entropy sampler that oversamples hard environment settings. The second is a "soft-risk" schedule that starts risk-neutral and anneals α toward the target. Both assume a simulator with explicit context parameters, and the schedule is a fixed function of training steps.

Risk-Averse Fine-tuning of Large Language Models (RA-RLHF) carries the soft-risk schedule to language models, but only on English data and with one global clock. Group-based policy optimization for reasoning with verifiable rewards makes the vanishing-gradient problem literal. A problem on which all sampled attempts fail has zero advantage and contributes nothing. Among the MGSM languages, this is most common for distant, low-resource ones.

Our claim has two parts. First, a closed-loop controller should adapt two quantities during training. One is a per-language risk level $\alpha_l$, driven by how flat that language's failure tail currently is. The other is a per-problem bridge fraction $\rho_x \in [0,1]$, the share of a problem's sentences shown in an anchor language. Together these should raise worst-language accuracy and cut the fraction of never-solved problems at a matched rollout budget, compared with a linear soft-risk schedule, language-balanced resampling, and worst-language optimization. Second, and this is the part that engages linguistic proximity, the choice of anchor should matter. A typologically or lexically close anchor should unlock more than an English anchor for languages that have a close relative in the pool.

This second part is a falsifiable test of a live tension in the literature. Cross-lingual Transfer of Reward Models in Multilingual Alignment and Language models are multilingual chain-of-thought reasoners argue that English is the best universal bridge. How Can Cross-lingual Knowledge Contribute Better to Fine-Grained Entity Typing? and Competence-based Curriculum Learning for Multilingual Machine Translation find that distant sources cause negative transfer and related ones help. If the English anchor does as well as the nearest cousin, we will report that proximity is irrelevant for this bridge. The adaptive controller would then stand on its own as a language-aware version of CeSoR.

We deliberately use verifiable rewards, meaning the answer is checked for correctness, rather than a learned reward model. M-RewardBench shows that reward models invert preferences on translated pairs and lose up to 13% accuracy off English. A CVaR objective computed on such a reward would preferentially train on the reward model's own mistakes in exactly the languages we care about. Exact-answer math removes that confound, so any tail improvement is a real one.

Research Program

Objective. The training objective for language $l$ is the CVaR, at level $\alpha_l$, of per-problem success probability. Let $p_\theta(x)$ be the fraction of $G$ sampled solutions to problem $x$ that are correct. For problems drawn from language $l$, we optimize

$$J_l(\theta) = \mathrm{CVaR}_{\alpha_l}\big[\,p_\theta(x)\,\big],$$

which is the mean success rate over the worst $\alpha_l$-fraction of problems. At $\alpha_l = 1$ this is ordinary average accuracy. In implementation, this follows RA-RLHF's batch filtering: within each language's slice of a batch, only the lowest-scoring $\alpha_l$-fraction of problems contribute group-normalized advantages.

Controller. Two quantities are updated in closed loop rather than on a timer. The per-language risk level $\alpha_l$ tracks a flatness statistic $f_l$, the fraction of that language's recent tail problems with zero correct samples. When $f_l$ is high, $\alpha_l$ relaxes toward 1, because there is no tail signal to exploit. When $f_l$ falls below a threshold, $\alpha_l$ steps toward the target level (0.2 in our main runs).

The per-problem bridge fraction $\rho_x$ is updated from that problem's own outcome:

$$\rho_x \leftarrow \mathrm{clip}\big(\rho_x + \eta\,(\mathbb{1}[p_\theta(x)=0] - \mathbb{1}[p_\theta(x)\ge \tau]),\,0,\,1\big),$$

where $\eta$ is a step size and $\tau$ a success threshold. The fraction grows when the problem fails entirely and shrinks once it is solved often enough. In the language of CeSoR, this is a feedback controller over a context parameter rather than a cross-entropy search. The difference is that the context here is linguistic and needs no simulator.

Bridged problems. A problem at bridge level $\rho_x$ replaces a $\rho_x$-fraction of its sentences with the anchor-language version, chosen so the question sentence stays in the target language. The model must always answer in the target language, and evaluation is always at $\rho = 0$, so the bridge cannot leak into test-time behavior.

Anchors. Anchors are chosen under three rules, which are the main comparison. The first rule picks the nearest related language in the pool by subword-vocabulary overlap, the same kind of proximity measure CCL-M uses to build its language graph. Examples are Spanish for Portuguese, Chinese for Japanese (shared characters), and Japanese for Korean. The second rule always uses English. The third picks a random pool language, as a control. A fourth, learned variant chooses the anchor by the measured alignment between the gradient on anchor-bridged problems and the gradient on the pure target problems. This lets us ask whether a measured transfer signal subsumes the typological prior.

Training data. The training set is GSM8K-train machine-translated into the target languages. P-MMEval reports machine-translation error rates up to 82.5% on technical content, so we keep only translations whose numbers exactly match the source. We also report results on the human-translated MGSM test set, so that translation noise in training does not inflate the evaluation.

Baselines

The do-nothing reference is the instruction-tuned starting model, Qwen3-1.7B and Qwen3-4B (the models MMPO uses), evaluated with native-language chain of thought. It anchors every gain.

The standard risk-neutral baseline is group relative policy optimization (GRPO) on the same multilingual problem pool, with uniform language sampling, the same group size $G=8$, and the same number of rollouts. This is the method whose zero-advantage groups motivate the proposal. It is fair when run with identical data, rollout budget, learning-rate sweep and number of updates.

The strongest existing risk-sensitive competitor is MMPO from Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning. MMPO minimizes the sum of the first $T$ moments of failure probability, which reweights toward hard problems. The paper reports a 2.6% average accuracy gain over GRPO with $T=4$ on English math (MATH500, OlymMATH, AMC23, AIME24, AIME25). There is no reported result on Lacuna for MMPO on MGSM or any multilingual benchmark. MMPO is the strongest competitor because it already targets hard problems without any language structure. If it matches us, the linguistic bridge adds nothing. We run it with $T=4$ and $G=8$, the settings the paper reports, under the same budget.

The strongest dynamic-risk competitor is RA-RLHF's linear soft-risk schedule from Risk-Averse Fine-tuning of Large Language Models, applied with one global α annealed over training steps. It gets the same tuning budget for its annealing length as our controller gets for $\eta$ and the flatness threshold. The paper reports gains in English generation tail reward (IMDB-Gen tail reward −1.11 against −1.51 for standard RLHF) but no multilingual reasoning number.

The strongest multilingual alignment reference is language-imbalance DPO from Language Imbalance Driven Rewarding for Multilingual Self-improving. It reports a 13.9% average MGSM accuracy improvement for Meta-Llama-3-8B-Instruct. We reproduce it on our backbone and also run our method on Llama-3-8B-Instruct, so that one comparison is on the model where a published number exists.

Two ablations isolate the key ingredients. The first keeps the per-language α controller but fixes $\rho = 0$, which removes the bridge. The second keeps the bridge but uses a fixed linear α, which removes closed-loop risk. Beating the baselines means two things at matched rollouts. The method must improve worst-language MGSM accuracy and lower the fraction of never-solved problems relative to both MMPO and the linear soft-risk schedule, by more than the seed-to-seed standard deviation across three seeds. It must also not lower the mean across languages by more than one point.

Why not just…?

Why not just reweight languages or run worst-language group optimization? Upweighting or minimax over languages, with the language holding the lowest current accuracy getting the most weight, is the most serious threat. It is simple and directly targets the worst language. It shares a blind spot with CVaR, though. Upweighting problems whose sampled groups all fail upweights zero-advantage terms, so it moves no parameters on exactly those problems. CCL-M's results against MultiDDS-S on TED translation, +1.21 to +1.42 BLEU, and the negative-transfer finding of Jin et al. both suggest that reweighting alone struggles with distant languages. Still, neither result is on reinforcement learning for reasoning. We will run worst-language minimax and temperature resampling at the same budget. If either matches our worst-language accuracy, then the bridge is not doing the work, and we will say so.

Why not just use MMPO, which already emphasizes hard problems? MMPO reweights by failure probability, but a problem with zero observed successes still gives it no usable gradient under group normalization. Its gains are also reported only in English. The ablation that removes the bridge while keeping risk adaptation, compared directly with MMPO, tells us whether the benefit comes from risk shaping, which MMPO could supply, or from moving problems off the flat plateau, which only the bridge supplies. The diagnostic is the fraction of zero-advantage groups per language over training.

Why not just translate to English, or reason in English? English chain of thought outperformed native reasoning on MGSM in Language models are multilingual chain-of-thought reasoners, where PaLM-540B reached a 55% average. This is a strong test-time baseline, and we report it. It does not, however, improve native-language competence, which is what P-MMEval's cross-lingual accuracy consistency ratio measures. It also inherits translation errors on technical content. Our English-anchor variant is the training-time version of this idea. If it ties the nearest-cousin anchor, the English-bridge view wins the Gap 4 argument for this setting.

Why not just scale the model? P-MMEval reports saturation in cross-lingual transfer with scale and persistently low accuracy for Thai and Japanese. BRIGHTER finds that 70B models fall below 30 F1 on Niger-Congo languages. We test at two sizes, 1.7B and 4B, to see whether the gain shrinks with scale. If it vanishes at 4B, the method matters only for small models.

Proposed Experiments

Main evaluation. We train on the translated GSM8K pool covering the MGSM languages and evaluate on the MGSM test set (250 human-translated problems per language). We also evaluate on the MGSM portion of P-MMEval, whose parallel design adds Arabic, Korean, Vietnamese and Portuguese and supports the cross-lingual accuracy consistency ratio. Metrics include:

  • mean accuracy;
  • worst-language accuracy;
  • the prompt-level CVaR at 0.2 of pass-rate over 16 samples;
  • the fraction of problems never solved in 16 samples;
  • the Gini coefficient of accuracy across languages, following MMPO;
  • the consistency ratio against English.

Decisive checks. Four results settle the proposal.

The first is mechanistic. During training, the bridge should cut the zero-advantage fraction in distant languages quickly, and that drop should precede the worst-language gains. If gains appear without the drop, our explanation is wrong.

The second is the anchor comparison of nearest cousin, English, random, and gradient-selected anchors, reported separately for languages that have a close relative in the pool (Portuguese, Japanese, Korean) and those that do not (Swahili, Telugu, Thai). Proximity is supported only if the cousin anchor beats English on the first group and the advantage disappears on the second.

The third is the head-to-head against worst-language minimax and MMPO at matched rollouts. This is the result that justifies the added machinery.

The fourth is a leakage check. At $\rho = 0$, the rate of anchor-language tokens in outputs must stay at the base model's level, so that gains are not bought with code-switched answers.

A negative outcome on the second check with a positive outcome on the third still yields a useful result: a language-aware, simulator-free version of soft-risk scheduling. A negative outcome on the third check means the problem is solved by reweighting, and the proposal should be dropped.