Bilingual Evidence Pools for Small Medical Question-Answering Readers

A small medical question-answering model that works well in English usually gets much worse when the same kind of question arrives in Arabic, Japanese, or Russian. Two remedies are available: pretrain the model on billions of tokens of medical text in the target language, or translate the question into English and hope nothing important is lost. This proposal tests a third option. Keep a small "reader" model and give it a mixed pool of evidence to read: background passages written in English by an English medical specialist model, where medical knowledge is densest, together with passages in the question's own language, which carry local drug names, guidelines, and exam conventions. The reader encodes each passage separately and combines them only when producing the answer. That combining step is the Fusion-in-Decoder (FiD) design, and it is what lets passages in different languages sit side by side without being merged into one long prompt.

The claim is deliberately narrow and can be falsified. On locale-neutral questions, plain translation into English should do about as well as the mixed pool. The mixed pool should help mainly on questions whose answer depends on local medical practice, which is exactly where translate-then-answer pipelines lose information. The work also tracks hallucination directly. Generated background passages can be wrong, so we ask whether disagreement between the English-generated and native-language passages warns us when the reader is about to follow a fabricated fact.

Core Claim

The closest existing system is To Generate or to Retrieve? (MedGENIE). It has an English medical generator (PMC-LLaMA-13B) write background contexts and fuses them with a 250M-parameter FiD reader, reaching 53.1% on MedQA. That beats 175B-parameter GPT-3/Codex, but only in English, and the authors acknowledge that the generator can invent clinical facts. On the cross-lingual side, MIA 2022 showed that moving from prompt concatenation to FiD was the largest single gain in a cross-lingual open-domain QA pipeline. That work covered Wikipedia trivia (XOR-TyDi, MKQA), with no medical domain and no measurement of hallucination.

Our claim is that a compact multilingual FiD reader fed a language-mixed evidence pool will beat two alternatives on non-English medical exams. The first is translate-test MedGENIE. The second is the same reader given an equally sized single-language pool. The improvement should be concentrated on locale-dependent questions. In addition, cross-language disagreement among the passages should predict reader errors better than the reader's own confidence does.

Mechanistic work motivates the design. Lost in Multilinguality finds that cross-lingual factual errors mostly arise when a model renders an answer it already knows into the target language, not from missing knowledge. Our pipeline puts knowledge generation in English, where the generator is strong. For multiple choice, the reader's job is then only to select among target-language options. CCL-XCoT also motivates the design: its purely parametric cross-lingual transfer cuts hallucination substantially but still struggles with dates and named entities, and the authors explicitly call for multilingual retrieval.

Research Program

We will build one pipeline and run four controlled variations of it.

The English evidence comes from PMC-LLaMA-13B. Following MedGENIE's recipe, it writes option-focused and option-free contexts from a machine translation of the question. The native-language evidence comes from two sources. One is passages retrieved from the multilingual medical corpus MMedC introduced by Towards building multilingual language model for medicine. The other is contexts generated in the target language by MMed-Llama 3 (8B).

The reader is mT5-base, a multilingual encoder-decoder. It is trained on the MMedBench training split, plus the MedAraBench training split for Arabic, and it always sees a fixed budget of K passages. During training we randomly drop all passages of one language with some probability. This "language dropout" keeps the decoder from learning to trust only English evidence, and it is the ingredient we ablate.

The analysis has three parts.

First, we annotate a stratified sample of test questions as locale-dependent or locale-neutral. Medical annotators do the labeling. A question counts as locale-dependent if it mentions region-specific drugs, regulations, epidemiology, traditional medicine, or national exam conventions. A classifier extends these labels to the full test set, and we report its agreement with the human labels.

Second, we measure context-following hallucination. This is the share of reader errors in which the chosen option was endorsed by at least one passage that a separate verifier judges wrong.

Third, we test cross-language agreement as an early warning signal. The agreement score is the fraction of passages, weighted by language, whose endorsed option matches the reader's answer. We evaluate it as a predictor of errors.

For free-form output, the reader also generates target-language rationales. These are scored against the 1,136 human-verified MMedBench rationales for unsupported claims, and also for response language correctness, meaning whether the answer comes out in the user's language. XRAG documents that this check fails often when English evidence is supplied.

Baselines

The do-nothing reference is the mT5-base reader fine-tuned on the same MMedBench and MedAraBench training data but given no passages. It shows what the evidence adds at all.

The first strong competitor is translate-test MedGENIE. We translate the question and options into English, run the unchanged English generator and the English Flan-T5 FiD reader, and map the chosen option back. MedGENIE's reported number is 53.1% accuracy on MedQA, in English. We found no reported result on Lacuna for translate-test MedGENIE on any non-English medical benchmark, so we run it ourselves. It gets the same generator, the same number of passages K, and an equal hyperparameter search budget. This is the baseline most likely to tie us, and the stratified analysis exists to locate where it does.

The second strong competitor is the parametric specialist MMed-Llama 3 (8B). The paper behind MMed-Llama 3 reports 67.75% average accuracy on MMedBench, against 62.79% for base Llama 3. For Arabic, the reference points from MedAraBench are 54.7% for Llama-3.3-70B-instruct, 32% for Llama-3.1-8B after QLoRA fine-tuning, and 76.5% for GPT-o3. We will not claim to beat MMed-Llama 3 with a reader roughly a fourteenth its size. Instead, the reader counts as competitive if it lands within about three points of MMed-Llama 3 on MMedBench at much lower inference cost, which we report as total FLOPs including the generator. It counts as better if it beats MMed-Llama 3 on Arabic, a language outside MMed-Llama 3's six languages.

The third competitor is MMed-Llama 3 given the identical mixed passage pool concatenated into its prompt. This separates "the evidence helps" from "FiD fusion helps."

Two ablations isolate our ingredients. The first gives the reader an English-only pool and a native-only pool, each of the same size K, which tests whether mixing languages matters. The second removes language dropout.

The headline result requires two things. The mixed-pool reader must beat translate-test MedGENIE and the English-only pool on locale-dependent questions, with a paired bootstrap confidence interval that excludes zero, and it must not lose on locale-neutral questions.

Why not just…?

Why not just translate the question and run English MedGENIE? This is the real threat. For multiple choice, translation errors matter less than in the open-retrieval span setting where XOR QA found translate-test pipelines capped near 18.7 F1. Our prediction is therefore conditional: translate-test should match us on locale-neutral items and fall behind only where English medical knowledge is wrong or silent about local practice. If it matches us on locale-dependent items too, we will report that the mixed pool is unnecessary for exam-style medical QA, and the project's contribution reduces to the hallucination diagnostics.

Why not just use a multilingual medical model like MMed-Llama 3? It is strong on its six pretraining languages, and we expect it to win there on raw accuracy. Two arguments favor our approach. First, it needed 25.5B tokens of target-language medical pretraining, which low-resource languages do not have. The Arabic results, a language outside its training set, test whether external evidence substitutes for that pretraining. Second, its parametric knowledge cannot be updated, while a passage pool can. The concatenated-pool variant shows whether MMed-Llama 3 itself benefits from the same evidence.

Why not just answer once per language and vote? Running the reader separately on the English pool and the native pool and averaging the option probabilities is cheap, and it captures much of what "using both languages" could mean. FiD's distinctive property is that the decoder can combine facts across passages. If voting matches joint fusion on locale-dependent questions, that property does not matter here, and we will say so. We add this vote as a further baseline.

Why not just fine-tune on translated MedQA? Teasing LLMs Adapted to Italian reports that fine-tuning on translated data degrades reasoning, and MedAraBench notes that translated benchmarks lose clinical nuance. We nonetheless include a reader trained with translated MedQA added to its training data, so the comparison is measured rather than assumed.

Proposed Experiments

The first decisive check is the stratified accuracy comparison. It covers the five non-English MMedBench languages and MedAraBench, comparing the mixed pool, translate-test, the single-language pools, and voting, with all results split into locale-dependent and locale-neutral questions. The prediction fails if the mixed pool's advantage is uniform across the two strata, or absent.

The second check tests whether cross-language agreement predicts errors. We measure the area under the ROC curve for agreement as an error detector, compared with the reader's maximum option probability. We then compute selective accuracy at 80% coverage, meaning accuracy when the system answers only the 80% of questions it trusts most. Agreement must add predictive value beyond the reader's own confidence to justify being called a hallucination signal.

The third check concerns generated rationales. We measure the unsupported-claim rate and response language correctness, and we compare English-heavy pools with the mixed pool. The point is to test whether English evidence reproduces the language-defaulting failure that XRAG documents.

The fourth check is a stress test. We plant a wrong English context for a subset of questions and measure how often the reader follows it, with native-language passages present and absent. If native passages do not reduce how often the reader follows the planted error, the claimed protection from cross-lingual corroboration does not hold.