Machine unlearning means removing specific knowledge from a trained model, such as a fictitious author's biography, copyrighted text, or hazardous biology, without retraining it from scratch and without breaking everything else. One of the cleanest recent methods works by making the model's internal attention less sharp. Attention Smoothing Unlearning (ASU) builds a "forget-teacher": a copy of the model whose self-attention softmax runs at a higher temperature, so each token spreads its focus more evenly over earlier tokens. Self-attention is the step where each token decides how much to look at every earlier token. The student model is then trained to imitate this blurrier teacher on the data to be forgotten. Sharp token-to-token lookups carry facts like names and dates, so flattening them erases recall while grammar mostly survives. The catch is that one scalar temperature flattens everything, including attention patterns that have nothing to do with facts. ASU therefore needs a careful search for the temperature, and the file lists this hyperparameter fragility as an open problem.
This proposal borrows a structural lesson from work on human information overload. When people receive more messages than they can process, they do not spread attention thinly across everything. They keep a protected core of recent and trusted items and ration whatever attention is left over (Gomez-Rodriguez et al.; Garibay). Transformers have an obvious analogue of that core: attention sinks, which are the first tokens that soak up leftover attention, and local previous-token windows, which carry syntax and position. We propose a forget-teacher that leaves this core exactly as it is. It then applies a budget cap only to the remaining long-range, content-selected attention, so that no single distant key can monopolize a query. Our claim is that most of ASU's collateral damage and temperature sensitivity comes from flattening the core. If that is right, a budgeted teacher should reach the same forgetting with a much wider range of acceptable hyperparameter values and less loss in fluency.
Core Claim
Raising the temperature does two things at once. It lowers the peak weight on the sharpest content key, which is the fact-retrieval pathway, and it changes how much total attention goes to the sink and local window. We hypothesize that only the first effect is needed for forgetting and that the second causes the loss in fluency and utility. The teacher we propose separates the two.
Fix a query position $i$ with attention weights $a_{ij}$ over keys $j$. Let $C_i$ be its core: the sink position(s) plus the $w$ most recent tokens. The teacher keeps $a_{ij}$ unchanged for $j \in C_i$. It then takes the leftover mass $m_i = \sum_{j \notin C_i} a_{ij}$ and redistributes it by projecting the non-core weights onto a capped simplex:
$$ \tilde a_{i\cdot} = \arg\min_{b} \sum_{j \notin C_i} (b_j - a_{ij})^2 \quad \text{s.t.} \quad \sum_{j \notin C_i} b_j = m_i,\;\; 0 \le b_j \le \kappa\, m_i . $$
Here $\kappa \in (0,1]$ is the largest share of the leftover budget that any single distant key may receive. The projection works like water-filling. It shaves the heaviest keys and passes the excess to the others in proportion, and it has a closed-form, sort-based solution. With $\kappa = 1$ the teacher is the original model. As $\kappa$ shrinks, the most "popular" distant keys are capped first. This loosely mirrors Garibay's finding that overloaded users stop deferring to globally popular accounts. We treat the social-science results as a design analogy that suggests which structure to protect, not as evidence about transformers. The transformer claim stands or falls on the experiments below.
Relation to Prior Work
ASU supplies the method we reuse. We keep its self-distillation setup, its retain-set regularizer, and its finding that early-to-mid layers (6–8 in Llama-2-7B) are enough. We also build on its observation that factual tokens are sensitive to attention sharpness while function tokens are not. Where we differ is the teacher: ASU scales all logits by $\tau$, while we fix the core and cap only the rest.
Two lines of work constrain the design. Negative Preference Optimization (NPO) shows that unbounded gradient ascent collapses into gibberish, and that a bounded objective slows collapse to a logarithmic rate. Our distillation target is also bounded, but NPO is a strong competitor on the same trade-off. WMDP's Representation Misdirection for Unlearning (RMU) pushes hidden activations on hazardous inputs toward a random vector. It shows that forgetting can resist GCG jailbreak attacks, and it sets the numeric bar for utility preservation on MMLU. An Illusion of Unlearning? warns that erased knowledge can often be recovered by light fine-tuning, so any gain in utility must be checked against relearning.
The overload literature motivates the protected-core idea. Gomez-Rodriguez et al. show that overloaded users preserve a stable core and filter the rest by recency. Huberman et al. show that real interaction concentrates on 4–13% of declared contacts. Neither paper says anything about neural networks.
We deliberately set aside the recommender-systems bridge the file suggests, in which unlearning methods would be made "attention-aware" of which clicks happened under overload. A deletion request must remove the requested interactions whether or not they reflected real preference. Discounting overloaded clicks would change the ranking model, not the unlearning problem.
Research Program
The work has three parts.
The first part is a diagnostic that could falsify the premise. On Llama-2-7B fine-tuned on TOFU, we apply ASU's temperature teacher at a sweep of $\tau$ values. For every head in layers 4–12, we record how much attention mass on sink and local-window positions changes, and separately how much the peak non-core weight changes. We then regress per-layer fluency loss and retain-set degradation on these two quantities. If core-mass change does not predict damage beyond what peak-weight change already explains, the central hypothesis is wrong and we stop.
The second part builds the budgeted teacher and trains students with ASU's exact loss and schedule, replacing only the teacher. Window size $w$ and the sink set are chosen once per model family from the diagnostic, not tuned per task, which leaves $\kappa$ as the only per-task knob.
The third part tests whether preserving the core reduces two known weaknesses. One is dependence on a retain set: we run with the retain loss removed, which is Gap 5 in the file and a real-world constraint the TOFU authors themselves flag. The other is drift under repeated unlearning requests over ten rounds.
Baselines
The naive reference points are the untouched fine-tuned model, which sets the utility ceiling and a forget quality of zero, and plain gradient ascent. Gradient ascent shows what collapse looks like; TOFU reports that such baselines "either failed to forget the data sufficiently or severely impacted the model's general performance."
NPO, run with its retain-set variant, is the strongest optimization-based competitor on TOFU. It is the only prior method reported to keep forget quality above p = 0.05 on Forget10 with high utility, and the first to give reasonable results at Forget50.
RMU is the strongest competitor on WMDP. On Zephyr-7B it cut WMDP-Bio accuracy from 63.7% to 31.2% while MMLU fell only from 58.1% to 57.1% (WMDP). Beating it means matching its WMDP-Bio accuracy within one point while losing less than its one MMLU point, or matching its MMLU while forgetting more.
ASU is the direct competitor. We run it at its reported default of $\tau = 2.3$ and also with its temperature tuned under the same search budget we give $\kappa$. ASU is reported to beat GA and NPO on TOFU with higher utility, but the file lists no reported numeric TOFU result for ASU. We will reproduce it rather than quote a figure.
Every method gets the same base checkpoints, forget and retain splits, number of optimizer steps, and a tuning budget of twelve evaluated configurations, scored on a held-out validation slice of the forget and retain sets.
Two ablations isolate the key ingredient. A "core-fixed temperature" teacher keeps the sink and window weights fixed but applies temperature scaling to the rest; this tests whether protecting the core alone does the work. A "cap without core" teacher applies the capped projection to all keys, including sinks and the local window; this tests whether the cap alone does the work.
Why not just…?
Why not just tune ASU's temperature per head? This is the most serious threat. With enough per-head temperatures, one could imitate core preservation by leaving syntax heads cold. The answer lies in how many settings must be searched. Per-head tuning over layers 6–8 of a 7B model means roughly a hundred temperatures, while our method has one parameter, $\kappa$. We will give per-head tuning the same twelve-configuration budget, using a greedy head-by-head search, and also report it with ten times that budget. If per-head tuning at the matched budget equals our frontier, our contribution shrinks to "a cheap parameterization of per-head temperature." We would report that honestly.
Why not just localize the fact-carrying heads by attribution and smooth only those? Weight-attribution approaches such as WAGLE confine updates to influential parameters. The difficulty is that fact-retrieval heads and syntax-bearing heads overlap, and the file identifies this parameter intertwining as the root cause of collapse. Head-level masking cannot separate a sink-attending pattern from a content lookup inside the same head. Our teacher acts within each attention row. We include a head-localized ASU baseline, with heads selected by gradient attribution on the forget set, to measure this directly.
Why not just use NPO or RMU and skip attention entirely? They may be enough on some benchmarks: RMU's WMDP numbers are strong, and NPO handles large forget fractions. Our claim is not that attention teachers dominate in general. It is narrower: within attention-smoothing unlearning, the damage comes from the core, and removing that damage makes the method less fragile to tune. If NPO or RMU beat our teacher on every benchmark at matched budget, the diagnostic is still a useful mechanistic result about ASU, but the method would not be worth adopting.
Proposed Experiments and Decisive Checks
On TOFU, we evaluate Forget01, Forget05, and Forget10 on Llama-2-7B, reporting forget quality (the p-value against a retrained model) and model utility. On WMDP-Bio and WMDP-Cyber with Zephyr-7B, we report accuracy together with MMLU. On MUSE we report the benchmark's verbatim and knowledge memorization measures. The file lists no reported MUSE numbers for any baseline, so all MUSE comparisons will come from our own runs.
The first decisive check is the diagnostic regression described above.
The second concerns robustness to the hyperparameter. For each method, we measure the fraction of its hyperparameter range that reaches TOFU forget quality with p above 0.05 while keeping utility within 5% of the original model. The prediction is that the acceptable range for $\kappa$ is at least twice as wide as for $\tau$. If it is not, the robustness claim fails.
The third is continual unlearning. Over ten sequential TOFU forget requests, we track utility after each round. ASU is reported to remain stable over ten rounds, so here we test for parity, not superiority.
The fourth tests retain-set dependence. With the retain loss removed, the prediction is that our teacher keeps substantially more utility than ASU because the core is never perturbed. The two ablations tell us which ingredient is responsible.
Finally, every model that gains utility is attacked by relearning, meaning a small amount of fine-tuning on related data, and by GCG. This ensures that preserving attention structure has not also preserved a path back to the forgotten knowledge.
References
- Attention Smoothing Is All You Need For Unlearning
- Quantifying Information Overload in Social Media and Its Impact on Social Contagions
- Inferring mechanisms of response prioritization on social media under information overload
- Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
- The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning
- An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations
- Social networks that matter: Twitter under the microscope
- TOFU: A Task of Fictitious Unlearning for LLMs