Repairing Your Own Mistakes: Self-Error Denoising to Unlock English-Bound Medical Knowledge in Other Languages

A medical language model often knows a fact in English and gets the same fact wrong in Swahili, Zulu, or Bangla. It answers fluently and confidently, and it names the wrong drug, dose, or disease. We propose to fix this with a training signal the model largely creates itself. We find the facts the model answers correctly in English but gets wrong in the target language. We take a clean target-language medical passage and paste the model's own wrong answer into it. Then we train the model to rewrite the corrupted passage back into the correct one. This is denoising auto-encoding (training a model to reconstruct clean text from a deliberately corrupted copy). Here the corruption is not random deletion or masking. It is the model's own characteristic cross-lingual hallucination.

The reason to expect this to work comes from interpretability. Lost in Multilinguality traced multilingual factual recall layer by layer and found that models retrieve the correct entity in a shared, English-leaning concept space in their middle layers. They then fail in the last two to four layers, where that concept is turned into target-language tokens. If this "final-layer trap" is the cause, the most useful supervision is an example where the model has just produced the wrong target-language surface form for a concept it already holds, paired with the right one. The central question is whether repairing a subset of such knowledge-locked facts teaches a general concept-to-language mapping that transfers to locked facts never seen during training. If it only patches the facts it trained on, the approach is ordinary fact memorization.

Core Claim

Continued training of a compact domain model (7–8B parameters) on a repair objective aimed at its own errors will close more of the English-versus-target-language gap on held-out medical facts than the three obvious alternatives: denoising with heuristic noise, entity masking, or preference training on the same error pairs. The model will also not become more prone to refusing questions. Two findings would each count as a result. One is transfer to held-out facts. The other is a measurable drop, visible through the logit lens (reading intermediate layers through the output head), in cases where the middle layer is right and the final output is wrong.

Position Relative to Prior Work

Three lines of prior work motivate the proposal. HealMed documents the problem: domain-specialized medical models such as MedGemma and MediPhi collapse in low-resource languages, and it reports a 15.4-point drop in Swahili and a 28.9-point drop in Zulu relative to English. Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models shows that making models larger raises per-language accuracy but not cross-lingual consistency, so scale is not the fix. Lost in Multilinguality identifies where the failure happens. It also shows that a linear map from middle-layer states to the output raises KLAR accuracy for LLaMA2-7B from 71.47% to 76.08%. That the fix is linear and generalizes across facts is the strongest hint that a learnable, general final-layer mapping exists. Their shortcut, however, only emits a single terminal token. We keep the model generative and push the repair into its weights.

Two lines of work supply the method. Sequence denoising is the method we reuse. BART and T5 established reconstruction from span-deleted or masked text. XNLG and XGLUE extended it across languages, including reconstructing a sentence in one language from its translation. All of these use noise that is blind to facts. We replace the noise function with the model's own error distribution. For plausible substitutes when the model happens to be right, we draw on medical term normalization tools, CODER and xMEN. These tools let us pick ontology siblings (for example, a drug in the same class) instead of random tokens.

The closest competitors constrain the design. CCL-XCoT aligns languages with contrastive continued pretraining on parallel text. It then has the model reason in English before translating its answer, which raises Tamil hallucination-free rates for LLaMA-3.1-8B to 55%, against 24% for LLaMA-3.1-70B-Instruct. Our approach differs in three ways: it does not pivot through English at inference, it uses a generative repair loss instead of a contrastive one, and its supervision is chosen by where the model actually fails. MedVL-XLRepE steers hidden states toward English at inference time. It needs white-box access and parallel calibration data at every deployment, and it leaves the weights unchanged. Finally, CMExam warns that small domain-tuned models learn to mimic clinical formatting while hallucinating entities. Any objective that rewards fluent reconstruction of a whole passage risks the same failure. For that reason we weight the loss toward the repaired spans and evaluate entity correctness directly.

Research Program

The data construction starts from clean medical passages in each target language. Where native text exists, we use it: Chinese CMExam solution explanations, and Bangla textbook material like the OCR'd corpora in BanglaMedQA. For Swahili and Zulu, native medical text is scarce, so we use machine-translated English medical text whose entity terms are normalized with xMEN and CODER. We treat this as a known weakness, because HealMed shows that machine-translation artifacts distort evaluation.

For each passage we mask one salient entity span and let the base model fill it, both in the target language and in an aligned English version. A span counts as knowledge-locked when the English fill is correct and the target-language fill is wrong. The corrupted training input is then the target-language passage with the model's own wrong fill inserted, and the output is the clean passage. When the model's target-language fill is correct, we instead insert an ontology sibling, so the model never learns that every passage is corrupted. Some passages are left uncorrupted for the same reason. The English passage never appears in the model's input. The model must pull the correct fact from its internal representation, which is what we want to strengthen.

Training uses a decoder-only backbone, Llama-3.1-8B or Qwen3-8B, with continued training on a denoising loss written as

$$\mathcal{L} = -\sum_{t} w_t \log p_\theta(x_t \mid \tilde{x}, x_{<t}),$$

where $x$ is the clean passage, $\tilde{x}$ is the corrupted passage, $x_t$ is the token at position $t$, $p_\theta$ is the model, and $w_t$ is a weight set larger on the repaired span than elsewhere. The weighting keeps the model from spending its capacity on copying fluent text. We mix in the model's original instruction data so that general behavior does not drift.

The analysis side has two parts. First, we split locked facts by entity into train and held-out sets. The held-out entities never appear in any corrupted or clean training passage in the target language, though the model may have seen them in English during pretraining. Second, we apply the logit-lens protocol of Lost in Multilinguality to held-out items. For each item we record whether the correct entity is decodable at the middle layer and whether it survives to the output, before and after training.

Baselines

The naive reference is the unmodified base model, together with the same model after plain continued pretraining on the same clean target-language passages with no corruption. Plain continued pretraining separates the value of the corruption from the value of simply seeing more target-language medical text.

The strongest existing method is CCL-XCoT (CCL-XCoT). We will re-implement it on the same backbone, using the same parallel medical text for its contrastive stage and the same instruction data. Its English-pivot inference is included, and we will report the extra tokens it generates. Its reported figure is a 55% hallucination-free rate in Tamil on CRAFT QA for LLaMA-3.1-8B. We found no reported result on Lacuna for CCL-XCoT on HealMed. Beating it means matching or exceeding its HealMed accuracy in low-resource languages without pivoting through English at inference, at comparable training compute. On HealMed itself, the reference point is the reported 21.7-point low-resource gap for Qwen3-32B-thinking; a smaller low-resource gap from an 8B model would be a meaningful result.

For bottleneck-free inference-time repair, we will run the linear shortcut of Lost in Multilinguality on KLAR, where it reports 76.08% average accuracy on LLaMA2-7B. We will also run MedVL-XLRepE-style steering vectors computed from the same parallel data. For retrieval, we will run traditional RAG over the same target-language corpus. The retrieval ceiling to beat is the reported 89.54% for agentic RAG with gpt-oss-120b on BanglaMedQA. We expect not to exceed that with an 8B model and no retrieval, and we will say so. The fair comparison is 8B plus RAG against 8B plus repair, and the two combined.

Three ablations isolate the key ingredient. We keep data, steps, and learning-rate sweep fixed and swap only the noise function among BART-style random span deletion, entity masking with infill, ontology-sibling swap alone, and self-error swap. A fourth ablation adds the English passage to the input, which turns the method into XNLG-style cross-lingual auto-encoding, to test whether withholding English is what forces the internal repair. Every method gets the same tuning budget of three learning rates and two data-mixing ratios, selected on a development split.

Why not just…?

Why not just mask the entities and have the model fill them in? This is the most serious threat. Salient-entity masking on the same passages may teach target-language recall just as well. The case for self-error swap is that the input contains a plausible wrong answer, which is the actual form hallucination takes at generation time: an entity swap, the failure mode reported in clinical records. Masking only teaches filling a blank. We have no evidence yet that this distinction matters. The noise-function ablation settles it directly. If masking matches self-error swap on held-out locked facts, we will report the simpler method as the finding.

Why not just train preferences on the same pairs, with the correct passage preferred over the one containing the model's error? Preference optimization uses exactly our pairs, so it is a real competitor. The difference is that repair conditions on the error being present in context and trains the model to map it to the right target-language form. Preference training only shifts probability between two complete continuations. We will run preference optimization on identical pairs with matched compute. The logit-lens check indicates which method actually reduces middle-right, final-wrong cases, as opposed to just lowering the likelihood of specific wrong strings.

Why not just translate the question to English, answer, and translate back? English-pivot pipelines can work, but HealMed shows that translation introduces terminology errors and that LLM judges reward translated phrasing. In XNLG, end-to-end generation beat the translation pipeline, 16.37 against 9.95 BLEU-4 on Chinese question generation. We will still run translate-test on HealMed's physician-reviewed items, because open-domain results may not carry over to medicine.

Why not just retrieve? On BanglaMMedBench, web search reduced accuracy on scenario questions, and CCL-XCoT notes that retrieval is unreliable in low-resource languages. Retrieval is complementary to repair, though, and we will report both on their own and combined.

Proposed Experiments

The primary benchmark is HealMed's 9-language, physician-reviewed set. We report accuracy on its multiple-choice and natural language inference tasks, and the gap in each language relative to English. For its open-QA items, we score answers with the benchmark's rubric, including the terminology dimension, where HealMed found problems in over 64% of Zulu responses. Secondary benchmarks are BanglaMedQA, BanglaMMedBench, and CMExam, the last of which reports 45.3% for fine-tuned ChatGLM-6B. For general factual consistency outside medicine, we use KLAR and BMLAMA with the RankC consistency metric.

To track the refusal trade-off, we use HalluLens PreciseWikiQA and NonExistentRefusal. HalluLens found that Llama-3.1-405B lowers hallucination to 26.84% but falsely refuses 56.77% of valid questions. A repair objective should cut hallucination without raising the false refusal rate. We also expect it to leave NonExistentRefusal unchanged, since repair does not teach abstention. Reporting that explicitly guards against overclaiming.

The decisive checks are three. First, on held-out locked medical facts, self-error repair must beat entity masking and preference training by a margin larger than the variation across three random seeds. Second, the logit lens must show that the fraction of held-out items that are correct at the middle layer and wrong at the output falls, while middle-layer correctness stays flat. That pattern would show the gain comes from the final-layer mapping, not from new knowledge. Third, the gain in Swahili and Zulu, where training text is machine-translated, must survive evaluation on HealMed's physician-corrected items; otherwise we are fitting translation artifacts.

If all three hold, a small, cheap, self-supervised stage turns an English-bound medical model into one whose knowledge carries across languages. If only the first holds, we have a useful recipe but not the mechanism we proposed. If entity masking wins, the literature gets a clean negative result about noise design.