Forgetting Once, in the Shared Middle: Cross-Lingual Unlearning That Survives Document Translation

When a multilingual language model has to forget something, such as a private person's biography, current methods usually make it forget in one language and then check a handful of others. The model is updated with gradient steps that push down the probability of the forgotten answers. Three recent findings make this approach look fragile. The knowledge often survives in other languages. The update mostly changes the last few layers, so the underlying facts can be recovered by adding a fixed direction to the hidden states at inference time. And the update can damage the model's ability to use the forgotten text when a user legitimately supplies it. This proposal tests a different design. We make a single small, low-rank edit to the middle layers. Those layers are where multilingual models are known to map different languages onto shared meaning. The edit is computed in closed form from English forget-set gradients. It is restricted to directions that are shared across languages, and it is made mathematically blind to activations produced when the entity appears in a supplied source document.

The test bed is document-level translation, and this choice is what makes the question new. Document translation is a setting where "forget it" and "still use it when given" collide in a sharp way. A translation system that has unlearned a person must still translate a document about that person correctly. It must also resolve "she" three sentences later to the right antecedent, so that German or Russian gets the right grammatical gender. At the same time it must not quietly insert memorized facts that the source never stated. No existing unlearning benchmark measures this. Existing multilingual unlearning work evaluates only isolated QA or probing sentences, and context-aware unlearning has only been studied for monolingual English retrieval-style QA.

Core Claim

Our claim has two parts. First, cross-lingual unlearning fails to transfer and stays reversible because the update is in the wrong place. We expect that one edit, confined to the language-shared subspace of middle layers, forgets in every language at once and resists inference-time steering recovery. Second, if that edit is projected onto the null space of "context-present" activations, the model keeps translating documents that contain the forgotten entity, including cross-sentence pronoun resolution. It does this at a cost of minutes rather than per-language fine-tuning runs. If the second part fails, that would itself be informative. It would mean that memory of an entity and processing of that entity in context share the same middle-layer directions, and so cannot be separated geometrically.

Where This Sits

The motivating evidence comes from three multilingual unlearning studies. LINGTEA combines gradient ascent with distillation from a pre-unlearning teacher and adaptive soft or hard labels. It shows that English-only unlearning leaves knowledge retrievable in other languages. Multilingual Amnesia runs GradDiff and NPO (Negative Preference Optimization, a loss that pushes forgotten answers below a reference model) on Aya-Expanse 8B across 10 languages. It finds that transfer tracks syntactic similarity, and that unlearning in lower-resource languages triggers catastrophic forgetting. Multilingual Unlearning in LLMs builds Multilingual TOFU, a set of fictitious author profiles in five languages. It shows that unlearning changes mainly the final layers, and that subtracting a "suppression direction" recovers 50% of forgotten knowledge in Qwen2.5 and 90% in Gemma2. All three update weights with unconstrained losses and evaluate on short QA. We replace the losses with one targeted geometric edit and evaluate on documents.

The case for the middle layers comes from representation analysis. Muller et al. show, using layer-similarity analysis and layer randomization on mBERT, that the lower and middle layers do the cross-lingual alignment. Xu, Li, and Xiong locate factual neurons in middle layers. They also show that shifting non-English representations toward English means (their method, LRP2) improves multilingual fact retrieval. That same paper also constrains our design: linear interventions are sensitive to which layer they are placed in. So layer choice is an experimental variable here, not an assumption. Two caveats apply to this argument. The mBERT result is about an encoder, so whether it carries over to decoder LLMs is part of what we test. And the late-layer finding was measured on facts injected by fine-tuning; whether such facts are stored in the middle layers at all is an empirical question that the edit-site sweep below addresses directly.

The method itself is borrowed. SEMU takes a singular value decomposition (SVD, a factorization into ranked directions) of forgetting gradients and modifies 0.5–1.8% of parameters. UNSC projects forgetting updates into the null space of retained activations, so that retained inputs are mathematically unaffected. On CIFAR-10 it drives forgotten-class accuracy to 0.70% while retained accuracy rises to 92.77%. Both were developed on vision models. Neither has been applied to multilingual language models, and neither has used a null space defined by in-context inputs. The context problem is defined by Forget to Know, Remember to Use. That paper shows that unlearning methods such as RMU collapse contextual QA to near zero. It restores contextual QA to 0.97–0.99 with a KL term that matches the original model's outputs only when the forgotten information is in the prompt. That work is English-only and QA-only, and its fix is a training loss. Ours is a geometric constraint, and we test it in translation across languages.

On the translation side, CorefCL and Context-Interactive Pre-training establish what a context-aware translator must get right. CorefCL targets cross-sentence coreference, measured on ContraPro. Context-Interactive Pre-training targets document coherence. Cont-COMET supplies a document-aware evaluation metric. We use these as the definition of what must be preserved, not as methods to compete with.

Research Program

The edit is computed per chosen weight matrix $W$ in a middle layer, using three ingredients. The first is $G$, the summed gradient of the forget loss on English forget-set questions. The second is $U$, an orthonormal basis for the language-shared output directions at that layer. We obtain $U$ from FLORES-200 parallel sentences by subtracting each language's mean activation (the "language identity" component) and keeping the top singular directions of what remains, which is what translations of the same sentence share. The third is $Q$, an orthonormal basis for "protected" input activations. These are collected from retain-set prompts and, critically, from source documents in which the forgotten entity is explicitly mentioned and must be translated. The edit is

$$\Delta W = -\eta \,\mathrm{SVD}_k\!\left[\,U U^\top G \,(I - Q Q^\top)\right],$$

where $\eta$ is a step size and $\mathrm{SVD}_k$ keeps the top-$k$ singular components. The $(I - QQ^\top)$ factor guarantees $\Delta W x = 0$ for every protected input $x$, following the logic of UNSC. The $UU^\top$ factor confines the change to cross-lingual meaning rather than language-specific surface form. If one pass is not enough, we iterate a few times with recomputed $G$. The total cost is a few backward passes plus SVDs, independent of the number of languages.

We build three things. The first is the edit procedure above, with the edit layer, the rank $k$, and the composition of $Q$ as swept variables. The second is a new evaluation suite, Multilingual TOFU-Doc. We expand each Multilingual TOFU author profile into short multi-sentence biographical documents in which the author is named once and then referred to by pronouns. Each document gets reference translations into German, Chinese, Russian, and Turkish, produced with machine translation followed by human post-editing. There are two document variants. In "fact-stated" documents the source states the facts, so faithful translation requires using them. In "fact-absent" documents the source omits them, so any appearance of a memorized fact in the output is leakage. The third is an analysis of where forgetting lives after each method. For this we reuse the layer-wise similarity and steering-recovery protocol of Multilingual Unlearning in LLMs.

Baselines

The naive reference point is the fine-tuned model before unlearning, together with a model retrained without the forget set. Together these bracket ideal forgetting from above and below on every metric. We also report a prompt-only baseline, a system instruction not to reveal the author, to anchor how little a black-box fix achieves.

The strongest unconstrained competitor is NPO and GradDiff applied to the forget set translated into all evaluated languages. This is the strongest version of the methods in Multilingual Amnesia and Multilingual Unlearning in LLMs. It is the most direct threat to the claim that a single shared edit is needed. We also run LINGTEA, the strongest published cross-lingual method. To compare against its published number, we reproduce its BLOOM-3B FLORES-200 memorization setting. There, LINGTEA reports forget-set Memorization Accuracy of 17.8%, against 35.8% for the original model and 25.5% for GradAscent+. Beating it means a lower forget-set memorization accuracy at equal or better retain perplexity. For context preservation, the strongest competitor is NPO plus the KL context term from Forget to Know, Remember to Use. It reports contextual QA of 0.97–0.99 in English. Beating it means matching its document-translation quality on fact-stated documents while being harder to reverse by steering. For the steering attack, the reference numbers are the 50% (Qwen2.5) and 90% (Gemma2) recovery rates. Lower recovery than every baseline at equal forgetting is the target. No reported result was found on Lacuna for any unlearning method on document-level translation or ContraPro, so those comparisons are new.

All methods use the same forget/retain splits (TOFU 1%, 5%, 10%), the same models (Qwen2.5-7B and Gemma2-9B, plus BLOOM-3B for the LINGTEA comparison), and the same hyperparameter search budget, measured in configurations tried. Selection is on a held-out validation split using the same combined criterion of forget quality and utility. We report GPU-hours for each method alongside quality.

Four ablations isolate the key ingredients. Removing $UU^\top$ tests whether language-shared targeting matters. Removing context activations from $Q$, keeping only retain activations, tests whether context protection matters. Applying the identical edit at late layers tests whether location matters. Finally, a plain SEMU-style SVD edit with neither projection tests whether we are just rediscovering low-rank unlearning.

Why not just…?

Why not just run NPO on the forget set translated into every language? This is the most serious alternative, and it may well match us on forgetting in the evaluated languages. The evidence against it is threefold. It costs one translated forget set and one tuning run per language, and still says nothing about unevaluated languages. Unconstrained updates in lower-resource languages cause catastrophic forgetting, according to Multilingual Amnesia. And such losses produce reversible late-layer suppression, according to Multilingual Unlearning in LLMs. Our experiments settle this in two ways. We include held-out languages never seen during unlearning, Hindi and Indonesian via FLORES-200 prompts. And we run the steering attack on both methods. If all-language NPO matches us on held-out languages and on steering recovery, the shared-subspace claim is false.

Why not just add the context-consistency KL term from Peng et al. to standard unlearning? That term already restores in-context use in English. It may transfer to translation. If it does, our null-space constraint is redundant for context preservation, and that ablation result will be reported as such. Two things could still favor our edit. One is that the KL term trains on prompts containing the entity, which may teach the model to use facts as a shortcut even when the source omits them. Our fact-absent documents measure exactly this leakage. The other is that the KL term still leaves the update in late layers, which the steering test probes.

Why not just shift all languages toward English with LRP2 and unlearn once in English? This is cheap and directly targets misalignment. However, Xu, Li, and Xiong report that the shift is fragile to layer placement and hurts low-resource languages. Their mLAMA gains are also modest, from 20.0% to 20.6% on mBERT. A shift applied at inference time also changes every input, which would distort document translation broadly rather than only for the forgotten entity. We include LRP2 plus English NPO as a baseline and measure its FLORES-200 translation quality cost.

Why not just filter outputs or prompt the model to refuse? This approach cannot pass any white-box check. The weights still contain the knowledge, so steering or a low-resource-language query bypasses the filter. Filtering also breaks fact-stated document translation, which by design must output the entity's details. The prompt-only baseline quantifies both failures.

Proposed Experiments

We fine-tune each model on Multilingual TOFU, then apply every method. Forgetting is measured with TOFU forget quality and truth ratio in all five Multilingual TOFU languages and in the two held-out languages. Robustness is measured as the fraction of forgotten answers recovered under the steering attack. Utility is measured with TOFU model utility, FLORES-200 sentence translation quality, and XNLI accuracy. Document translation is measured on Multilingual TOFU-Doc with Cont-COMET, as calibrated by Hu et al. with three preceding and three following sentences. To this we add a coreference score: the accuracy of gendered pronoun translation for the forgotten author in German and Russian. Generic pronoun ability is checked on ContraPro, to confirm the edit does not damage coreference for anyone else. Leakage is the rate at which held-out memorized facts appear in translations of fact-absent documents.

The work has four decisive checks. The first is whether the middle-layer shared edit lowers steering recovery well below 50% on Qwen2.5 and 90% on Gemma2 at forgetting equal to NPO. The second is whether forgetting reaches held-out languages without per-language data. The third is whether the edit keeps fact-stated document translation and author-pronoun accuracy within noise of the pre-unlearning model, while leakage on fact-absent documents falls to the retrained-model level. The fourth is whether all of this happens at a small fraction of all-language NPO's GPU-hours. The layer sweep is itself a result. If forgetting and context use separate cleanly in the middle layers, the edit should work. If they occupy the same directions, the context-protected edit will fail to forget, and we will have located the place where "forget" and "use when given" cannot be separated by a linear edit.