When we ask a generative model to forget something, such as a person, a copyrighted style, or an outdated fact, we usually judge success by whether the forbidden thing stops appearing in the output. Many current methods meet that bar by teaching the model a new, active behavior: produce noise, produce maximal randomness, or produce something wrong whenever the forgotten concept is requested. A model that genuinely never learned a concept behaves differently. It produces plausible, generic content that simply lacks the concept. When it is shown the concept, it learns it about as quickly as it would learn any new thing. Uncertainty decomposition gives these two behaviors names. Splitting a model's uncertainty into the part caused by inherent ambiguity in the data (aleatoric) and the part caused by the model's own lack of knowledge (epistemic) shows that noise-style forgetting raises the wrong kind. The practical obstacle has been that nobody knows what the "right amount" of ignorance looks like for a large generative model, because the exact reference, a model retrained without the forgotten data, is too expensive to build.
Data that changes over time supplies that reference at no cost. In any time-stamped corpus, such as monthly news or yearly arXiv abstracts, new entities, topics, and terms keep appearing. A model trained up to a cutoff has never seen the concepts that emerge after it. Its uncertainty on those future concepts is real ignorance, measured on the same kind of concept and the same domain as whatever we want it to forget. This note proposes to use those naturally unseen future concepts as the calibration target for generative unlearning. Forgetting a concept should drive the model's epistemic uncertainty on that concept up to the level it shows on genuinely unseen future concepts of the same type, while its aleatoric uncertainty stays where it was. For concepts that themselves emerged late, a checkpoint saved before they appeared serves as an actual retrain-equivalent, which lets us check the whole procedure against ground truth.
Core Claim
The claim has three parts. First, a decomposed uncertainty signature (epistemic, aleatoric) separates three outcomes that output-level metrics treat as the same:
- Fake forgetting: epistemic uncertainty stays low.
- Noise forgetting: aleatoric uncertainty is inflated.
- Genuine forgetting: epistemic uncertainty matches that of a model that never saw the concept, and aleatoric uncertainty is unchanged.
Second, temporal corpora supply a matched pool of truly unseen concepts, from which the target signature can be estimated without retraining. Third, an unlearning objective that matches this signature will sit closer to the retrain reference than either entropy maximization or total-uncertainty matching. We measure closeness by output behavior, by how quickly the concept is relearned, and by how much it rebounds when training continues on new data.
The novelty lies in pairing the decomposed target with the free, temporally matched ignorance reference. Neither idea alone is new. LoTUS already matches forget-set outputs to the uncertainty of unseen held-out data. Evaluating Machine Unlearning via Epistemic Uncertainty already argues that epistemic uncertainty is the right yardstick.
Where This Sits in Prior Work
Three methods motivate the proposal by showing what goes wrong with treating uncertainty as a single number:
- SAFEMax maximizes the output entropy of a diffusion model on forget classes, so that it denoises into Gaussian noise. It reports that oracle classifiers sometimes read that noise as a specific class, which suggests the model learned a mapping rather than returning to ignorance.
- Orthogonal Entropy Unlearning similarly maximizes forget entropy in classifiers. It also contributes a projection that removes the component of the forgetting gradient that conflicts with the retain gradient, and we reuse that projection.
- LoTUS matches total output uncertainty to unseen samples in classifiers.
All three target total entropy. None asks which kind of uncertainty was raised, and none works with generative language models or with drifting data.
Three papers constrain the design by showing that output metrics lie:
- An Illusion of Unlearning? finds that linear probes recover roughly 70–80% accuracy on classes whose output accuracy was driven to zero.
- Tackling Fake Forgetting through Uncertainty Quantification finds that over half of "forgotten" points keep their true label inside conformal prediction sets. Its conformal guarantee assumes exchangeable data, which temporal drift breaks.
- UnlearnCanvas shows that sequential unlearning in diffusion models rebounds, and that jailbreak prompts drop unlearning accuracy from nearly 100% to below 50%.
These results mean the evaluation must include relearning speed, probing, and rebound, not just output suppression.
Three sources supply the measurement tools:
- Language models. We use the standard ensemble decomposition behind disagreement-based methods such as BALD-SAM. For cheaper single-model estimates we use Contextual Similarity Distillation.
- Image generators. We use PUNC, which captions generated images with a vision-language model and scores two quantities against the prompt. One minus recall (prompt concepts missing from the image) serves as epistemic uncertainty. One minus precision (extra or hallucinated content) serves as aleatoric uncertainty. PUNC has never been applied to unlearned models.
The temporal side does three things. It confirms that a simple baseline is hard to beat: Wild-Time and Temporal Generalization: A Reality Check both find that plain training on all past data, or simply using the latest checkpoint, beats elaborate robustness methods. It also provides the closest existing combination of unlearning and drift. That is UIL, which unlearns the oldest chunk of a sliding window instead of retraining. UIL is discriminative, uses no uncertainty measurement, and struggles on semantic drift. Finally, it raises the confound we must control: Guo et al. separate calibration decay from loss of discrimination over time. In our setting, rising uncertainty could come from drift rather than from forgetting.
Research Program
Generative model and data. The main testbed is a family of small autoregressive language models trained month by month on the NewsRoom corpus (2012–2016), using the Temporal Generalization codebase. A second stream comes from the arXiv titles and abstracts in Wild-Time (2007–2022), treated as a text-generation corpus.
Concepts. From each stream we extract named entities and topical terms with a known first appearance. Concepts that appear after a model's cutoff form its unseen pool. We draw forget concepts from two groups. Late-emerging concepts have a pre-emergence checkpoint that serves as a retrain reference. Long-established concepts have no such checkpoint.
Measuring the decomposition. For language models we fit $K$ lightweight adapter heads on a shared backbone, which gives an ensemble with members $p_k$. On prompts about a concept $c$, total uncertainty splits as
$$ \mathcal{H}\Big[\tfrac{1}{K}\textstyle\sum_k p_k(y\mid x)\Big] \;=\; \underbrace{\tfrac{1}{K}\textstyle\sum_k \mathcal{H}[p_k(y\mid x)]}_{U_a(c)} \;+\; \underbrace{\mathrm{MI}(y;k\mid x)}_{U_e(c)}, $$
where:
- $\mathcal{H}$ is entropy over next-token or continuation distributions;
- $U_a$ is the expected per-member entropy, which we treat as aleatoric;
- $U_e$ is the mutual information between the output and the member index, which we treat as epistemic.
We add a second, ensemble-free epistemic probe based on hint sensitivity. If the model genuinely lacks a concept, its perplexity should drop sharply once a short defining passage is placed in context. Ambiguity that is aleatoric should not shrink this way.
Unlearning objective. The objective has three terms: a retain loss on the current window, a term that pulls the forget-concept epistemic score toward a target $u_e^\star$, and a term that holds the aleatoric score at its pre-unlearning level $u_a^{0}$:
$$ \mathcal{L}(\theta) = \mathcal{L}_{\text{retain}}(\theta) + \lambda_e\big(U_e^{\theta}(D_f) - u_e^\star\big)^2 + \lambda_a\big(U_a^{\theta}(D_f) - u_a^{0}\big)^2 . $$
The target $u_e^\star$ is the median epistemic score on unseen-pool concepts matched to the forget concept by type (person, organization, or term) and by how often it appeared in the training data. In the streaming setting the unseen pool is refreshed every window, so the target follows the drift and does not compare against a stale reference. We remove the part of each forgetting gradient that conflicts with the retain gradient, following the orthogonal projection of OEU.
Analysis. The analysis has two halves. One is diagnostic: we place every existing method on the epistemic–aleatoric plane next to the retrain reference and the unseen pool. The other is causal: we ask whether matching that signature buys anything beyond suppressing the output.
Images. On UnlearnCanvas with Stable Diffusion v1.5, we run PUNC on styles unlearned by ESD, UCE, SalUn, and a SAFEMax-style entropy objective. We expect noise-style methods to show inflated aleatoric scores (low precision), while suppression methods leave recall partially intact. A small decade-conditioned diffusion model trained on Wild-Time Yearbook portraits extends the temporal reference idea to images: a model trained up to 1970 has genuinely never seen 1990s portraits.
Baselines
Naive reference points. There are three:
- No unlearning: the model as trained, which gives the ceiling on leakage.
- Latest checkpoint: the model trained on the current window, which already forgets old material passively. Temporal Generalization: A Reality Check found this the strongest temporal baseline. That paper gives no numeric values in the file, and no NewsRoom unlearning result was found on Lacuna.
- Full retraining from scratch on the window: the gold standard, affordable only at our small scale.
The pre-emergence checkpoint plays the role of the retrain reference for late concepts.
Strongest existing method: total-uncertainty matching. This is a LoTUS-style procedure adapted to generation: temper outputs on forget prompts so their total entropy matches unseen data, with no decomposition. It is the closest competitor because it already uses "look like unseen data" as its target. LoTUS reports 94% retain accuracy on ImageNet1k and a 0.92 correlation between its retrain-free JSD (Jensen–Shannon divergence) and retrained JSD (LoTUS). For language-model generation, no reported result was found on Lacuna.
Entropy maximization. This is SAFEMax's objective adapted to next-token distributions. SAFEMax reports unlearning accuracy "often reaching 100%" on DDPM CIFAR-10 (SAFEMax).
Gradient ascent with retain regularization. This is the standard simple unlearning recipe.
Sliding-window unlearning. For the streaming experiments we run UIL-style influence unlearning of the oldest chunk. UIL reports recovery from MNIST noise drift in 6.33 chunks versus 17 for sliding-window retraining (UIL).
Ablations. Two ablations isolate the key ingredient:
- drop the aleatoric-preservation term, keeping only epistemic matching;
- replace the temporally matched unseen pool with a random set of rare held-out tokens, which tests whether temporal matching matters.
Fairness. Every method starts from the same checkpoint and sees the same forget and retain data. Each gets the same number of gradient steps and the same budget of hyperparameter configurations, tuned on a validation month disjoint from the test months.
What counts as beating a baseline. Our method must move closer to the retrain reference than total-uncertainty matching does, on the combined distance defined below, without lower retain performance. It must also hold up on long-established concepts, where no direct reference exists.
Why not just…?
Why not just distill the pre-emergence checkpoint on forget prompts? Where such a checkpoint exists this is a real threat, and it may match or beat our objective. We include it as an upper-bound baseline for late concepts. It cannot be used for concepts that were already present at the start of the stream, or for copyrighted material that was always in the data. Those are most real deletion requests. The decisive test is whether a signature target calibrated on late concepts transfers to early concepts, judged against full retraining at small scale. If it does not transfer, the contribution reduces to an evaluation tool, and we will say so.
Why not just match total uncertainty to unseen data, as LoTUS does? Total entropy can be matched equally well by a model that became uncertain and by one that learned to emit diffuse but structured junk. The decomposition argument predicts the two differ in relearning speed and probe accuracy. The first ablation, compared against the LoTUS-style baseline, tests exactly this. If the two end up equally close to retraining, decomposition adds nothing as an objective. It would still matter as a diagnostic, given the conformal and probing evidence from Tackling Fake Forgetting and An Illusion of Unlearning?.
Why not just let the sliding window forget passively, or retrain on it? Passive forgetting cannot remove a concept that is still present in recent data, and a deletion request rarely aligns with a window boundary. Retraining is infeasible at foundation-model scale. At our scale we can afford retraining, so it serves as the reference rather than a competitor. We still report latest-checkpoint leakage on forget concepts, because if it is already near the reference, the problem is smaller than claimed.
Why not just maximize entropy, since it already achieves near-perfect unlearning accuracy? Unlearning accuracy is the metric we doubt. We predict entropy maximization will sit off the reference in the aleatoric direction, which PUNC precision on UnlearnCanvas and the language-model decomposition can both show. It would also relearn or rebound differently from a model that never knew the concept.
Proposed Experiments and Decisive Checks
Primary metric. The primary metric is distance to the retrain reference, combining three quantities:
- the divergence between output distributions on forget prompts;
- relearning speed, the number of gradient steps on a few examples of the concept needed to restore its likelihood, compared with the reference;
- linear-probe accuracy for concept presence in hidden states.
Secondary metrics. These are retain perplexity on the current window, perplexity on future months (to check for drift damage), and rebound. Rebound is measured as follows: after unlearning, training continues on four further monthly chunks that do not mention the concept, and we track whether its likelihood recovers, as it does in UnlearnCanvas's sequential setting. We report jailbreak-style paraphrase prompts separately.
Check 1: diagnostic power. Does position on the epistemic–aleatoric plane predict relearning speed and probe accuracy better than unlearning accuracy or total entropy does, across all methods and concepts? If it does not, the premise fails.
Check 2: transfer. On late concepts the full objective should approach the pre-emergence checkpoint. On early concepts, judged against full retraining, it should still beat total-uncertainty matching on combined distance.
Check 3: drift confound. Freezing the unseen pool at an old window should degrade results over time, while refreshing it should not. This would show that the temporal matching does real work, and that the method is not mistaking drift-induced uncertainty for forgetting.
The image experiments act as a cross-check on mechanism, not a headline. We expect noise-style unlearned styles on UnlearnCanvas to show elevated one-minus-precision, and the Yearbook model trained up to 1970 to show elevated one-minus-recall on 1990s prompts. If the image results reverse that pattern, the noise-versus-ignorance account is wrong for diffusion models, and the claim narrows to language models.
References
- LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty
- Evaluating Machine Unlearning via Epistemic Uncertainty
- Unleashing Uncertainty: Efficient Machine Unlearning for Generative AI
- Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks
- An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations
- Tackling Fake Forgetting through Uncertainty Quantification
- UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models
- BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation
- Contextual Similarity Distillation: Ensemble Uncertainties with a Single Model
- Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation
- Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
- Temporal Generalization: A Reality Check
- Unlearning-based sliding window for continual learning under concept drift
- Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine