Do Cross-Lingual Knowledge Edits Reach Native Speakers?

When a fact changes, such as a new minister being appointed or a club changing coaches, the usual fix for a language model is a knowledge edit: a targeted update that makes the model give the new answer without retraining it. A multilingual model edited in English should then give the new answer in Hindi, Arabic or Swahili too. Several benchmarks measure this. Every one of them (Bi-ZsRE, MzsRE, CROLIN-MQUAKE, BMIKE-53) builds its non-English test questions by machine-translating English ones. Machine-translated text is a recognizable dialect of its own, often called translationese. It keeps the source's word order, word choices and phrasing. A multilingual encoder can tell it apart from native text more than 83% of the time. So the field's main measure of whether an update "reaches" other languages is taken on text that mirrors the English edit far more closely than any real speaker's question would.

We propose to measure cross-lingual knowledge editing on questions written by native speakers, for real facts that have actually gone out of date. We pair each native question with the machine-translated version of the same question. Our central prediction has two parts. First, translated benchmarks overestimate how far edits travel, because translated questions share surface wording with the English edit. Second, the missing transfer on native questions does not show up as random error. It shows up as the model confidently reverting to the outdated fact. This is the cross-lingual form of what RealTime QA calls the "outdated trap." If both hold, knowledge obsolescence in non-English languages is worse than current numbers suggest, in a specific and measurable way.

Core Claim

The claim is testable on a single matched benchmark. For the same fact change and the same edit, the drop in edit success from a translated question to a native question of equal meaning, which we call the native–translated gap, is positive for current editors. A large share of the failures it adds are stale reversions: the model outputs the pre-edit value. Two lines of evidence motivate this.

The first comes from cross-lingual transfer. Translation Artifacts in Cross-lingual Transfer Learning showed that translation changes surface cues such as word overlap between sentences. Gains on translated XNLI partly come from matching those artifacts. On original, untranslated Spanish data, the ranking of methods even reversed. Bridging the Gap between Native Text and Translated Text through Adversarial Learning showed that XLM-R separates native from translated text with over 83% accuracy. It also showed that transfer to native event-extraction text improves sharply once that difference is removed.

The second comes from editing itself. Cross-Lingual Knowledge Editing in Large Language Models found that edits behave like changes to surface strings rather than concepts: SERAC generality falls from 50% in English to 19% in Chinese. If edits are string-bound even on translationese, which sits close to English, native phrasing should strain them further.

The obsolescence half supplies the failure mode we expect. RealTime QA found that when retrieval fails, the model falls back on stale knowledge stored in its weights instead of abstaining.

We define the gap for an editor $E$ and language $\ell$ as

$$\Delta_{E,\ell} = \mathrm{Gen}_{E,\ell}^{\text{translated}} - \mathrm{Gen}_{E,\ell}^{\text{native}},$$

where $\mathrm{Gen}$ is the fraction of meaning-equivalent questions answered with the edited value. We report alongside it the stale-reversion rate: the fraction of questions answered with the pre-edit value.

Research Program

Benchmark. The first component is a matched benchmark of real fact changes. We start from real-world updates whose old and new values are both known: the Wikidata-sourced WikiFactDiff split used in BMIKE-53 and the temporal subset CROLIN-MQUAKE-T from Cross-Lingual Multi-Hop Knowledge Editing. We focus on six CROLIN-MQUAKE languages that span Latin and non-Latin scripts: Spanish, German, Russian, Chinese, Hindi and Swahili.

For each fact we collect three question forms. The first is the benchmark's existing machine-translated question. The second is written by a native speaker who sees only the subject, the relation and an example answer, never the English sentence, so the phrasing is not anchored to English. The third is a native-register question modeled on how the topic is discussed in local news or social posts. This third form tests the formal-to-informal genre shift that IITD at WANLP 2022 found blocks transfer from English news to Arabic tweets.

We also vary where the edit comes from. In one condition it is an English triple-style statement, as in current benchmarks. In the other it is a native-language local news sentence reporting the change, which is how such updates actually reach the world. Two annotators verify that the question forms are equivalent in meaning. As a quality check, we confirm that an XLM-R discriminator separates our translated and native questions at rates similar to the 83% Yu et al. report.

Measurement study. The second component runs the main editor families on this benchmark: locate-then-edit (ROME, MEMIT), memory-based (SERAC, IKE), instruction-tuned synchronization (X-KDE), and retrieval-based (CLEVER-CKE). For each, we measure the gap, the stale-reversion rate, locality, and the language-confusion rate that BMIKE-53 identified, meaning answers given in English instead of the target language. We then test the proposed cause. We regress edit success on the word and embedding overlap between question and edit statement, controlling for language and question form. If overlap explains the gap, the bias is mechanical. If the gap persists at matched overlap, something deeper, such as concept alignment, is failing.

Remedy. The third component tests a deliberately simple fix that targets the identified cause. When an edit arrives, a multilingual LLM writes several native-style restatements of the new fact directly in each target language, prompted as "how a local outlet would report this," not as translations. These go into the editor's training signal: the paraphrase set for SERAC and X-KDE-style tuning, or the stored entries for CLEVER-CKE's memory. This borrows the generate-then-filter approach of Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations, including its diversity filtering on LaBSE embeddings. The difference is that we apply it to edit propagation instead of task training data.

Baselines

The naive reference point is the unedited model. Its accuracy on the new value and its rate of giving the old value fix the floor and calibrate the stale-reversion measure. The second naive reference is the edited model scored only on the translated questions. This is what the field currently reports, and the proposal's premise is that it is the misleading number.

The strongest existing system for real temporal multi-hop updates is CLEVER-CKE from Cross-Lingual Multi-Hop Knowledge Editing. It keeps the LLM frozen and stores edits in a multilingual memory searched by a contrastively trained retriever. It reports up to 30% higher accuracy than MeLLo and PokeMQA on CROLIN-MQUAKE. In the same paper, ROME and MEMIT stay below 5%. It is the right main competitor because its authors found retrieval to be the bottleneck, and native phrasing should hurt retrieval most.

For single-hop edits, the strongest method is X-KDE from Edit Once, Update Everywhere, which reports a +8.19% average improvement over LTE on Bi-ZsRE. IKE is evaluated under BMIKE-53's best setting, 8-shot metric-specific demonstrations, which raised Llama-3.1-8B zsRE portability from about 10.0 to 32.8.

No reported result was found on Lacuna for any editor on native-written questions. That absence is the gap the benchmark fills.

For fairness, all methods use the same backbones (Llama-3.1-8B, plus Llama-2-7B for comparability with CLEVER-CKE), the same edits, and the same evaluation questions. Each method gets the same hyperparameter search budget, tuned on a held-out set of translated questions, so that native questions stay fully unseen.

The key ablation separates our ingredient from simply adding more data. We compare native-style restatements against an equal number of machine-translated paraphrases of the same edit. Only the provenance of the text differs. The remedy beats a baseline only if it shrinks the gap and the stale-reversion rate on native and native-register questions without lowering translated-question accuracy or locality. Gains limited to translated questions would count against it.

Why not just…?

Why not just translate native questions into English and query the model edited in English? This translate-test pipeline avoids cross-lingual transfer entirely and is a real threat. Artetxe et al. found translate-test competitive once its artifacts are handled. We run it as a full baseline using the same MT system for every language. If it closes the native gap, that is itself a publishable conclusion: knowledge updates should be routed through a pivot language rather than edited into a multilingual model. Two things could still count against it. One is its accuracy on native-register questions, where MT of informal text degrades. The other is whether the model's answer comes back in the right target-language form, an issue for transliterated names in Hindi and Russian.

Why not just back-translate the edit statements, as Artetxe et al. did for training data? Back-translation injects translationese into training. That should help on translated benchmarks and is exactly the wrong target for native questions. Artetxe et al.'s own original-Spanish result shows models matched to translationese do worse on natural text. The matched-provenance ablation above tests this directly.

Why not just store the edit in every language and rely on retrieval? This is essentially CLEVER-CKE plus translated memory entries. If the retriever's failures on native questions come from surface mismatch, translated entries will not fix them. RealTime QA's finding that 34% of errors are retrieval failures followed by stale fallback predicts how this fails. Our stale-reversion analysis, split by whether the correct entry was retrieved, measures this directly rather than assuming it.

Why not just adversarially align native and translated representations, as Trans+Adv did? That method trains an encoder for a classification head. Applying it to a generative 8B editor is far more expensive, and it is not obviously suited to edits. We treat it as follow-up work, worth pursuing only if the cheap remedy fails while the overlap regression implicates representation-level mismatch.

Decisive Checks

The study turns on four results. The first is whether the gap is reliably positive across editors and languages, with confidence intervals bootstrapped over facts. The second is a difficulty control: native and translated questions must be equally hard before editing, meaning the unedited model must give the old value at similar rates on both. Otherwise the gap reflects question difficulty, not translationese. The third is whether added native-question failures are mostly stale reversions rather than off-target answers. That pattern would tie the result to obsolescence rather than to generic noise. The fourth is whether the gap grows from Latin to non-Latin scripts, beyond the script effect already seen on translated data. Cross-lingual Editing in Multilingual Language Models reports generability of 90.04% within Latin languages versus 78.38% in Indic ones.

The study would also produce a negative result worth reporting. If the gap is near zero after the difficulty control, translated editing benchmarks are vindicated, and the remaining failures lie in concept alignment rather than domain mismatch.