Multilingual language models often solve a math word problem better if they reason in English, or in several languages at once, and then vote on the answer. But languages are not equally reliable voters. When the prompt contains irrelevant sentences, or the question depends on local cultural knowledge, some languages become much worse and others hold up. Current cross-lingual voting gives every language one equal vote and breaks down as weak languages are added. This proposal treats each language as an annotator of unknown skill. It estimates each language's accuracy from how often it agrees with the other languages across a stream of unlabeled test questions, then weights the votes accordingly. No labels, no gradient updates and no offline topic-to-language table are needed.
In two sentences: if two independent annotators agree on a free-form numeric answer, they are almost certainly both right, so pairwise agreement rates over many unlabeled questions reveal each annotator's accuracy. We apply this idea, borrowed from classic crowdsourcing models such as Dawid–Skene, to reasoning chains produced in different languages. The resulting weights re-estimate themselves when the test distribution shifts, for example when distracting context appears. The weights also expose pairs of languages that fail together, which is exactly the failure that makes naive voting degrade.
Core Claim
Self-consistency decoding means sampling many reasoning chains and returning the most common final answer. Its value depends on how independent and how accurate the chains are, not on how many there are. Cross-Lingual Consistency (CLC) translates a problem into several languages and majority-votes across them. It reports gains of 4.1% to 18.5% on the Multilingual Grade School Math benchmark (MGSM) with Gemma2-9B-Instruct. It also reports that accuracy declines beyond about six languages, a failure it calls Probabilistic Consensus Degradation, and it offers no remedy. Language Specific Knowledge shows that for cultural questions a single native "expert" language beats majority voting. That paper finds the expert language through an offline topic-to-language mapping.
We claim that both findings are symptoms of one problem: votes are equally weighted even though annotator reliability varies. We further claim that reliability can be recovered without labels, from the test stream itself. Concretely, let $A_{ij}$ be the fraction of test questions on which language $i$'s answer matches language $j$'s answer. Let $p_i$ be language $i$'s unknown accuracy on this stream. Suppose errors are independent and wrong free-form answers rarely coincide. Then two languages agree essentially only when both are right:
$$A_{ij} \approx p_i p_j + c_{ij},$$
Here $c_{ij}$ is the excess agreement caused by correlated errors, and it is zero under independence. We fit $p$ by least squares on the off-diagonal entries, a rank-one fit. Each language then votes with weight
$$w_i = \log \frac{p_i}{1 - p_i},$$
This is the log-odds weight that is optimal for independent annotators. The same fit is useful diagnostically. Large residuals $c_{ij}$ flag language pairs that share failure modes. Examples would be two languages whose translations came from the same flawed pivot, or two languages that are both fooled by the same distractor. Such pairs can be merged into a single effective vote.
The central empirical claim is about distracting context. Distractors, meaning irrelevant but plausible sentences inserted into the problem, change which languages are reliable, and the change is not uniform across languages. Breaking Focus shows that such distractors cut the accuracy of mainstream models by over 45%. For example, GPT-4o-mini on CommonsenseQA fell from 85.7% to 22.0%. It also shows that telling the model to ignore irrelevant information does not help. The research file's own candidate-gaps section notes that nobody has measured how this vulnerability interacts with language. Weights fitted on the live stream should track a shift like this, while fixed weights or equal votes cannot.
Research Program
The method is a gradient-free form of test-time adaptation, meaning the system adjusts to a shifted test distribution using only the unlabeled test inputs. It is closer in spirit to the memory caches of TDA than to parameter updates. The system works in five steps.
- For each incoming question, the model translates it into a pool of $L$ languages, using itself as the translator in the style of self-translate. The original language and English are always included.
- It samples $k$ reasoning chains per language, and each language's internal majority becomes that language's single vote. Samples within one language are highly correlated, so they are collapsed rather than counted separately.
- The agreement matrix $A$ is updated online.
- $p$ is refit and the weighted vote is taken.
- The first few dozen questions fall back to equal weights until the estimates stabilize.
To capture the pattern that different topics favor different languages, reported by Language Specific Knowledge, we also fit reliabilities within clusters of question embeddings. These cluster-level estimates are shrunk toward the global estimate when a cluster is small. For multiple-choice or short-answer cultural tasks, wrong answers collide more often, so the agreement model gains a chance term equal to the probability that two wrong answers coincide by accident.
Three analyses come alongside the method.
- Distraction by language. We measure the drop from distractors separately for each language. This fills the "compound vulnerability" gap directly, because it tests whether low-resource languages are hit harder, and whether reasoning through English partially filters out distractors written in the native language.
- Error correlation. We measure the true error correlation across languages on labeled data, which checks whether errors become more or less independent under distraction.
- Estimator recovery. We check whether the unlabeled estimates of $p_i$ match accuracy measured with labels.
The recovery check is the load-bearing one. If the rank-one fit does not recover accuracy, the method has no basis.
Prior work supplies the parts but not the combination. DiSCTT uses self-consistency agreement to decide which test questions to train on. It is monolingual and assumes clean inputs. We use agreement across languages instead, and to weight votes rather than to drive gradient updates. LiRA anchors low-resource languages to English through fine-tuning on parallel data. We change no parameters. Cross-Lingual-Thought prompting and multilingual chain-of-thought motivate English as a strong anchor. Our method does not assume English is best. It measures whether English is best on the current stream.
Baselines
The naive reference point is greedy native-language chain-of-thought, meaning a single reasoning chain produced deterministically in the question's own language. Next to it is greedy English chain-of-thought. Multilingual chain-of-thought found English chains consistently better than native ones on MGSM, and PaLM-540B averaged 55% there. Self-translate reports 46.6% versus 43.1% for direct inference with LLaMA-30B, averaged across tasks. These baselines set the floor and show how much sampling buys.
The strongest competitor on clean math is English-only self-consistency at matched compute. It exploits the known English advantage and pays no translation cost. For it, and for all other voting methods, we fix the total number of generated tokens per question, including translation tokens, at a budget roughly equal to 24 chains. The English baseline therefore gets more chains than any single language in the multilingual methods. No reported result was found on Lacuna for English-only self-consistency on MGSM with inserted distractors, so we will establish that number ourselves.
The strongest existing cross-lingual method is CLC (Cross-Lingual Consistency). It reports gains of 4.1% to 18.5% on MGSM with Gemma2-9B-Instruct, depending on the language set, and a 9.5% absolute gain on CMATH with DeepSeek-Math-7B-Instruct. We will rerun it with the same model, the same language pool and the same budget, and sweep $L$ from 2 to 12. Beating CLC means two things: matching or exceeding its best accuracy at $L \le 6$, and showing no decline as $L$ grows past 6.
For cultural questions the strongest competitor is LSKExtractor (Language Specific Knowledge). It reports up to 10.4% relative improvement over the best baselines on CultureAtlas and outperforms majority voting on BLEnD and Social IQa. We run it with its offline mapping built on the same data it was designed for. We count it as matched if our label-free cluster weights come within its confidence interval.
Two ablations isolate what the method adds. The first is confidence-weighted voting, where each language is weighted by how often its own samples agree with each other. The second is a supervised upper bound, where fixed per-language weights are fit on a labeled clean development split. The first tests whether agreement across languages, rather than self-agreement, is what matters. The second tests whether estimating on the live stream matters once the distribution shifts.
Why not just…?
Why not just run English self-consistency with more samples? This is the most serious threat, and on clean MGSM it may win. English chains are strong, and the budget spent on translation instead buys more English samples. CLC's reported advantage over monolingual voting suggests that single-language sampling tends to repeat the same mistake. However, CLC compared against native monolingual voting, not against English voting at matched budget. The file contains no direct evidence against English-only voting, so our matched-budget comparison settles the question. We commit in advance to reporting clean-setting results where English self-consistency ties or wins. Our claim rests on the distracted and cultural settings, where English's reliability should drop relative to other languages.
Why not just weight languages by a small labeled development set? Fixed weights would probably match our method when test and development data come from the same distribution. In that case our only advantage would be not needing labels, which is a real but modest contribution. The case for our method is shift. Weights fit on clean MGSM should be wrong once distractors appear, because distraction changes per-language accuracy unevenly. The experiment calibrates on clean data and tests on distracted data. If fixed clean weights still match stream-fit weights, the adaptation claim fails and we will say so.
Why not just weight each language by its own sample agreement? This baseline is label-free and nearly free to compute. However, self-agreement measures consistency, not correctness. The direction-1 limitation noted in the file, that systematic errors are reinforced rather than washed out by voting, predicts that languages with confident systematic errors would receive high weight. The ablation tests this directly. If self-agreement weights recover labeled accuracy as well as the cross-language rank-one fit, the more complex estimator is not needed.
Why not just filter the distractors or instruct the model to ignore them? Breaking Focus reports that such instructions fail, and that its fix, preference training on pairs of distracted and clean reasoning, requires training. DAS handles irrelevant retrieved documents through abstention prompts and query-likelihood reweighting, but only in single-pass English. We include an "ignore irrelevant information" instruction in every arm, so any gain from our method comes on top of it.
Proposed Experiments
We build MGSM-Distract by inserting semantically plausible irrelevant sentences into each of MGSM's 250 problems in all 10 languages. We use two variants. In the first, English distractors are generated in the style of Breaking Focus and then translated. In the second, distractors are written natively in the problem's language. Comparing the two separates distraction from translation artifacts.
The benchmarks are MGSM and CMATH in clean and distracted form, plus BLEnD, CultureAtlas and Social IQa for the cultural setting. We run two open models used in prior work, Gemma2-9B-Instruct and DeepSeek-Math-7B-Instruct, plus one larger open model. The metric is exact-match accuracy with bootstrap confidence intervals.
Four checks decide whether the idea works.
- Estimator validity. Across models and settings, the unlabeled $p_i$ should correlate with labeled per-language accuracy, with a rank correlation of at least 0.8. If it falls below 0.5, the method is abandoned.
- Adaptation under shift. The fitted weights should reorder between clean and distracted streams. On MGSM-Distract, weighted voting should beat both English self-consistency and CLC at matched budget, while fixed weights calibrated on clean data should fall behind.
- No degradation with more languages. Accuracy should stay flat or rise as $L$ grows from 6 to 12, where CLC declines. The residuals $c_{ij}$ should single out the language pairs responsible for the decline.
- Cultural questions. Cluster-level weights should assign the highest weight to the native language on culture-specific clusters and to English on arithmetic, and should match LSKExtractor without its offline mapping.
As a secondary measurement, the per-language distraction drops on MGSM-Distract give the first characterization of how distraction interacts with language. This measurement is useful even if the voting method fails.
References
- Cross-Lingual Consistency: A Novel Inference Framework for Advancing Reasoning in Large Language Models
- Language Specific Knowledge: Do Models Know Better in X than in English?
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Efficient Test-Time Adaptation of Vision-Language Models
- Do Multilingual Language Models Think Better in English?
- DiSCTT: Consensus-Guided Self-Curriculum for Efficient Test-Time Adaptation in Reasoning
- Toward Robust Multilingual Adaptation of LLMs for Low-Resource Languages
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
- Language models are multilingual chain-of-thought reasoners
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering