Scientific language models are trained on corpora that contain far more than papers. Journal mastheads, conference front matter, committee rosters and memorial tributes are full of researcher names tied to administrative roles, such as "Associate Editor of IEEE Transactions on Fuzzy Systems." Suppose we want a model to forget these documents. The difficulty is that the same names also appear as authors of real research the model should keep. We propose to use this situation as a natural, realistic test of machine unlearning, meaning methods that remove the influence of specific training data from a model without retraining it. The test asks whether a method can delete one kind of fact about a person (their editorial or committee role) without deleting another kind of fact about the same person (what they published and worked on). It also asks whether the deletion survives when the model later trains on new papers by those same people.
Our central prediction is that today's robust unlearning methods fail in a specific and measurable way here. They will damage authorship knowledge for people named in the forgotten documents more than for unrelated people. They will also see suppressed role facts come back under "benign" continued training, but only when that new training data shares entities with the forgotten documents. If both effects hold, the fix is cheap: build the retain set, the data the model is told to preserve during unlearning, from research facts about the very entities that appear in the forget set. Testing that fix is the methodological contribution.
Core Claim
The unlearning literature already treats relearning as the central robustness threat. Beyond Sharp Minima shows that gradient ascent leaves the model in narrow loss basins that small weight changes undo. Unlearning That Lasts shows that fine-tuning on unrelated data can restore forgotten facts, and that a bounded Jensen-Shannon objective (JensUn) makes forgetting much harder to reverse. In both works the relearning data is either the target itself or something unrelated to it.
We claim there is a third, more realistic regime that neither paper tests. Here the follow-up training data is legitimately different from the forget set but shares its entities. A scientific model that continues pretraining on new papers will see the names from old mastheads again. Our hypothesis is that this entity-overlapping training revives role facts much more than entity-disjoint training does. It does so by re-strengthening the shared internal representation of the person.
The motivating evidence comes from adjacent work. In diffusion models, Co-occurring Associated REtained concepts in Diffusion Unlearning (ReCARE) shows that erasing a target such as "nudity" destroys co-occurring benign concepts such as "person" unless those concepts are explicitly protected. It reports a retention score of 0.94 for its own method against 0.11 for the STEREO baseline. In scientific text, Can Large Language Models Truly Forget Scientific Claims? (SciUnlearn) shows that heavily cited, often-repeated papers resist unlearning, which it calls the embeddedness factor. Mastheads repeat in every issue of a journal, so they give us a natural dose variable for embeddedness.
How This Differs From the Closest Work
SciUnlearn is the nearest study. It works in the same corpus, Dolma, with the same model family, OLMo. It unlearns 219 scientific claims and finds that paraphrase generalization nearly fails: gradient difference lowers accuracy on the trained forget queries from 97.54% to 71.70%, but on unseen paraphrases only from 93.92% to 93.51%. Its forget units are claims, it never touches non-research documents, and it does not separate facts that share an entity.
Machine Unlearning for Document Classification unlearns a whole document class, "letter," from a ResNet image classifier on RVL-CDIP. It has no text, no entities, and no relearning audit.
ReCARE supplies the core idea of a retain set built from co-occurring concepts. However, it builds that set by clustering CLIP image-text embeddings for diffusion models. We instead build it from the corpus's own authorship structure, and we test it under relearning, which ReCARE does not.
The forget documents themselves are ordinary catalog entries. Examples are IEEE Transactions on Fuzzy Systems publication information, a masthead listing editors and society officers; DSML 2023 Committee, a workshop roster; and Past IJCAI Conferences, a list of venues, years and chairs. These have exactly the dense name-role structure we need.
Research Program
Gold-model protocol. The position paper on the overuse of "machine unlearning" argues that unlearning claims mean little without comparison to a model retrained without the forget data, and we adopt that standard. Retraining OLMo from scratch is out of reach, so we make the gold model affordable through continued pretraining.
We start from a public intermediate OLMo-1B checkpoint. We continue pretraining it on a scientific mixture drawn from Dolma in two versions: one including a set of identified non-research documents, and one identical except that those documents are removed. The second run is the retrain-without-forget-set reference for everything learned during that stage. To make the dose variable controllable, the injected documents keep their natural repetition: a masthead appears once per issue it was printed in.
Identifying the documents. We find non-research documents with the structural signals the literature describes: a very high ratio of names to tokens, roles such as "Editor-in-Chief" and "Program Committee," and no experimental language. Every forget document is then verified by hand. Detection is a means here, not a contribution.
Probe sets. For each forget document we write role probes, for example "Who served as associate editor of X in year Y?" or "What role did Z hold for conference C?" Each probe has 15 paraphrases plus in-context variants. We score free-form answers with a language-model judge and take the worst case over paraphrases, following JensUn. This avoids both the multiple-choice illusion described in LLM Unlearning Under the Microscope, where a model learns to pick wrong options while still generating the forbidden content, and the blind spots of word-overlap scores.
We also build two retention probe sets from research papers in the same mixture. The entangled set covers authorship and research-topic facts about people named in the forget documents. The disjoint set covers the same kinds of facts about matched people who appear in no forget document, matched by paper count and citation count.
The method. For each forget document we extract its people and venues. We then assemble an entity-matched retain set: research-paper passages and question-answer pairs about those same entities, taken from the retained corpus. This set replaces or supplements the generic retain set in an existing robust unlearning objective. We pair it with two methods that the literature disagrees about. One is JensUn, a bounded-divergence method. The other is RMU, a representation method that pushes the model's internal activations on forget data toward noise. LLM Unlearning Under the Microscope reports that representation methods resist benign fine-tuning better, while Unlearning That Lasts reports that bounded divergence nearly eliminates reversal. Our setting tests which claim holds when relearning shares entities with the forget set.
Relearning and stress tests. After unlearning, we fine-tune on held-out new scientific papers in two conditions with equal token counts and equal topic mix. In the first, the papers contain the forget-set entities. In the second, they contain none of them. We also run a small direct relearning attack on forget documents, following the 80-sample attack used for StableUN, and apply 4-bit quantization, which LLM Unlearning Under the Microscope identifies as a passive recovery route.
Scale check. We finally repeat the main comparisons, without a gold model, on OLMo-3-7B-Instruct. This is the model SciUnlearn used, which tells us whether the effects persist at scale.
Baselines
The naive reference points are the original model before unlearning and the gold model trained without the forget documents. Every forgetting and retention number is reported relative to both. A method "forgets" only to the extent that it moves role-probe accuracy toward the gold model's level, which may be above zero if role facts are also stated in research papers. A method "preserves" only to the extent that entangled-retention accuracy stays near the gold model's.
The strongest existing competitor for durable forgetting is JensUn from Unlearning That Lasts. It reports 0% worst-case forget accuracy on its Lesser-Known Facts benchmark while keeping response quality close to the original model, and it explicitly targets benign relearning. We run it with its generic retain set, matching the size of our entity-matched set. We use the same steps, learning-rate sweep and evaluation probes, and select hyperparameters on a validation split of probes disjoint from the test probes.
The strongest competitor for resisting weight perturbations is StableUN from Beyond Sharp Minima. It reports an average 14.55% robustness improvement over gradient ascent with a KL penalty under 80-sample relearning attacks on WMDP-bio. We run it under the same budget.
RMU with a generic retain set is the representation-level competitor. We also include NPO and gradient difference, because SciUnlearn's numbers on Dolma-derived claims give a calibrated reference for how weak they are. That paper also reports MMLU falling from about 58 to about 39 for NPO without a retain objective.
FLAT from LLM Unlearning via Loss Adjustment with Only Forget Data is the forget-only reference, run with the authors' released code. Why Forget-Only Unlearning Needs Memorization proves that forget-only deletion cannot guarantee both utility and indistinguishability from a retrained model when deleted and kept data are hard to tell apart. Shared entities are a concrete case of that, so FLAT is where we expect the largest entangled collateral damage.
The ablation that isolates our ingredient holds the objective fixed (JensUn or RMU) and varies only the retain set. The three versions are generic, entity-matched, and entity-matched but with the same number of tokens drawn from disjoint entities. The third controls for the possibility that the benefit comes from scientific-domain retain data rather than entity overlap.
No reported result was found on Lacuna for unlearning non-research scholarly documents with any method. Our numbers on this testbed will therefore be the first. "Beating" JensUn means two things at once. First, the entangled-minus-disjoint retention gap must shrink to within noise of the gold model's gap, where JensUn with generic retain shows a significant gap. Second, worst-case role-probe accuracy after entity-overlapping relearning must be lower, with forgetting before relearning no worse. In both cases we require confidence intervals over at least three seeds and over forget-set resamples.
Why not just…?
Why not just filter these documents out and retrain? For future models this is the right answer, and our gold model is exactly this. The proposal is not an argument for unlearning mastheads in production. It argues that these documents form an unusually clean testbed. They are real pretraining data, cheap to retrain around at small scale, low in harm, and naturally tangled with retained data through shared names. That combination is missing from TOFU's synthetic biographies and from hazardous-knowledge benchmarks, where retraining comparisons are rarely run. What we learn about entity-entangled deletion carries over to the cases that matter, such as removing personal information about someone who also appears in legitimate retained text, where filter-and-retrain is not available after release.
Why not just add a refusal behavior or system-prompt guard? Refusal-style methods teach a surface mask over particular phrasings, as LLM Unlearning Under the Microscope documents. Unlearning That Lasts shows that measured knowledge on Llama-3-3B rises from 46% to 72% when worst-case paraphrases replace a single query. We include a refusal-tuned model as a baseline, and the worst-case paraphrase evaluation will show directly whether role facts leak. We expect that it will.
Why not just use JensUn or RMU with any reasonable retain set? This is the real threat. If generic scientific retain data already protects entangled authorship facts and blocks entity-overlapping relearning, our method adds nothing. The retain-set ablation settles this. If generic and entity-matched retain sets produce the same entangled-minus-disjoint gap, we will report that robust unlearning already handles entity entanglement. That would still be a useful negative result, because it would contrast with ReCARE's diffusion-model finding.
Why not rely on continued training on clean papers to wash the documents out? Unlearning That Lasts finds that such training tends to restore suppressed facts rather than erase them. Kang et al. find retain-only fine-tuning slow at erasure, though only for an image classifier. Our relearning arm tests this directly, since the entity-overlapping condition is exactly the continued scientific pretraining a reviewer would propose.
Decisive Checks
Four results decide the proposal.
The first is the collateral gap. With a generic retain set, does unlearning lower authorship accuracy for forget-set people significantly more than for matched disjoint people, relative to the gold model? If not, the entanglement premise fails.
The second is the relearning asymmetry. Under equal-token fine-tuning, does entity-overlapping data revive worst-case role-probe accuracy more than entity-disjoint data, for each of JensUn, RMU and StableUN? This is the new robustness claim, and a null result would mean that prior benign-relearning findings already describe the whole phenomenon.
The third is whether entity-matched retain closes the gap. It should bring both the collateral gap and the relearning asymmetry toward gold-model levels without raising forget accuracy, and it should outperform the token-matched disjoint control.
The fourth is the dose response. Residual role-probe accuracy should rise with the number of issues in which a masthead appeared, mirroring SciUnlearn's embeddedness factor. For the forget-only FLAT baseline, membership-inference distinguishability from the gold model should be highest for entangled entities, which is what the deletion-gap argument predicts. Throughout, MMLU, GSM8K and IFEval track general capability, and 4-bit quantization serves as a final check that any forgetting we report is not a fragile boundary that disappears when weights are rounded.
References
- Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimization
- Unlearning That Lasts: Utility-Preserving, Robust, and almost Irreversible Forgetting in LLMs
- Co-occurring Associated REtained concepts in Diffusion Unlearning
- Can Large Language Models Truly Forget Scientific Claims? A Systematic Evaluation of Claim Level Unlearning
- Machine Unlearning for Document Classification
- IEEE Transactions on Fuzzy Systems publication information
- DSML 2023 Committee
- Past IJCAI Conferences
- Position: The Term “Machine Unlearning” Is Overused in LLMs
- LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
- LLM Unlearning via Loss Adjustment with Only Forget Data
- Why Forget-Only Unlearning Needs Memorization