When someone asks a speech system to forget their voice, the model is usually updated to push its predictions away from that person's recordings. Real recordings, however, are not pure voice. They mix the speaker with the room, the street, the wind, or a hum of machinery. If a person's clips share a distinctive acoustic background, an unlearning update cannot tell which part of the clip it is supposed to erase. It can take the easy route and learn to reject the background. The forgetting test then looks successful, because the person's clips from that background are no longer recognized. Meanwhile the voice itself may still be identifiable when it turns up in a different environment, and other people recorded in the same environment pay the price.
This note proposes to measure that failure and to fix it. The fix has two parts. The first is a forgetting signal built from the person's voice re-mixed into many unrelated backgrounds. The second is a protection step that blocks any update aimed at features the model uses for background sound. The central claim is testable: standard audio unlearning, evaluated the usual way on the original recordings, overstates how much a voice was forgotten. A cross-environment forgetting test exposes the gap, and environment-aware unlearning closes it at a fraction of the cost of retraining.
Core Claim
Machine unlearning means updating a trained model so it behaves as if certain training data had never been seen, without retraining from scratch. In audio, the obvious version is already published. Speech Unlearning adapts gradient ascent, random relabeling, and saliency-based unlearning to Whisper, wav2vec 2.0, and HuBERT on Speech Commands and VoxCeleb1. It adds a curriculum-style reweighting loss, and it reports two failures. Gradient ascent collapses the model: forget-set accuracy falls to 12.3% while overall test accuracy falls to 7.1%. Membership inference scores also stay at 40–50%. A membership inference attack is an attacker guessing whether a given recording was in the training set.
That paper names "feature-level unlearning" as open. It describes the problem as an "entanglement trap": speaker, phonetic content, and acoustic environment are encoded together. It never varies the acoustic environment, so it cannot tell whether its residual leakage comes from background shortcuts. Unlearning LLM-Based Speech Recognition Models studies memorization of clean synthetic speech inserted into LibriSpeech. It suggests noisy prefixes as a stress test but leaves the idea untested.
Our proposal differs from both in what it treats as the unit of forgetting. Their unit is the recording. Ours is the speaker's voice, separated from the acoustic scene it was captured in. Our evaluation also differs: it asks whether forgetting survives a change of environment.
The mechanism comes from two places. The idea that background and foreground can be pulled apart inside a network is motivated by Spatio-Temporal Attention Pooling for Audio Scene Classification. That paper separates transient foreground events from static background bands using separate time and frequency attention and a background-subtracted input channel. The protection step reuses the projection idea of Projected-Gradient Unlearning. PGU confines unlearning updates to directions that matter little for retained data, where "retained data" means the training data the model should keep. We change which directions are protected.
The suspicion that unlearning hides rather than erases comes from Erase or Hide?. That paper shows that gradient ascent in language models creates inhibitor neurons that suppress the forgotten knowledge, which leaks back after brief fine-tuning. A stationary background band is an ideal target for such an inhibitor in audio. An example is the persistent 20–300 Hz shipping drone documented in Comparison of Two Soundscapes.
Research Program
The first component is a confounded benchmark, which current corpora lack. We take clean multi-speaker speech from LibriSpeech and VCTK and assign each speaker a "home" acoustic scene. The scenes are real environmental recordings from the 19 categories of LITIS Rouen. We mix each speaker's speech with their home scene across a range of signal-to-noise ratios, where low ratios mean the background is loud relative to the voice. Some retained speakers share the forget speakers' home scenes.
We then train a speaker-identification head on top of a self-supervised backbone (wav2vec 2.0 and HuBERT, as in the speech unlearning paper). We also hold out every speaker's speech mixed into foreign scenes. This gives us two test conditions. In-environment forgetting is the standard test. Cross-environment forgetting asks whether the voice is still recognized when the room changes. The difference between them is the "environment shortcut gap," which is our main diagnostic. We repeat the construction on VoxCeleb1, where the backgrounds are naturally confounded with speakers because they come from real interview recordings. There we cannot control the scene, but we can stratify by estimated signal-to-noise ratio.
The second component is the method. Let $s_f$ be a forget speaker's clean speech and $\{n_k\}$ a bank of backgrounds that excludes the home scene. We compute the forgetting gradient $g_f$ on the re-mixes $s_f + n_k$. Because the background varies across re-mixes while the voice stays fixed, the only consistent signal in $g_f$ is the voice.
We then protect background features. At each layer, we collect activations on background-only clips (home scenes with no speech) and on retained speech. We take their top singular vectors to form a protected subspace with basis $U$, and apply the update
$$\Delta\theta = -\eta\,(I - UU^\top)\,g_f,$$
where $\eta$ is the step size and $(I - UU^\top)$ removes any component of the update lying in the protected subspace.
The difference from PGU is that $U$ explicitly includes background-only activations. PGU uses only retained-data activations, so it protects a background only to the extent that retained data happens to cover it. The forgetting loss itself follows PGU: reverse cross-entropy plus maximizing prediction entropy on the forget clips.
The third component is the attack suite. It has three parts:
- the speaker-level and utterance-level membership attacks from Membership Inference Attacks Against Self-supervised Speech Models, run on both in-environment and foreign-environment queries;
- a relearning attack in the spirit of Erase or Hide?, consisting of brief fine-tuning on a small fraction of forget data;
- a probe that measures whether forgotten speaker identity is still linearly decodable from intermediate backbone features.
Baselines
The naive reference is doing nothing: the original model, which gives the ceiling of leakage.
The gold standard is retraining from scratch without the forget speakers, on identical data and recipe. Every metric is reported as a distance to this retrained model. Its wall-clock time sets the cost scale.
Gradient ascent and random relabeling are run as in Speech Unlearning. Its reported numbers on Speech Commands are the reference failure mode a correct reimplementation should reproduce: forget accuracy 12.3%, overall test accuracy 7.1%, and membership scores of 40–50%. That paper's curriculum-reweighted SuperLoss strategy is the strongest published audio-specific method. The file gives no isolated number for it, so we rerun it ourselves.
The strongest general competitor is PGU, applied to the same backbone with its retained-data protected subspace. On CIFAR-10, PGU reports a membership inference AUC of 0.5340 with retained accuracy within 0.01% of retraining, at 11.6x speedup. No reported result was found on Lacuna for PGU on any audio benchmark. We also run Boundary Unlearning, which shifts output decision boundaries and is a pure output-layer method. Because it never touches backbone features, we expect it to forget in-environment while leaving identity decodable.
All methods receive the same forget and retain sets, the same backbone checkpoint, the same gradient-step budget, and the same hyperparameter search budget, tuned on a validation split of speakers disjoint from those reported.
Two ablations isolate the key ingredients. The first uses re-mixed forget clips with the plain PGU subspace, with no background protection. The second uses background protection with the original, un-remixed forget clips.
Beating PGU means satisfying three conditions together:
- cross-environment forget accuracy and membership AUC within a small tolerance of the retrained model;
- retained accuracy on same-scene speakers within the same tolerance;
- a smaller environment shortcut gap.
Each condition must hold at matched compute, across five random forget-speaker draws.
Why not just…?
Why not just denoise the audio before training and unlearning? If enhancement removed the background, the entanglement would disappear. The evidence says it does not at realistic signal-to-noise ratios. In Text-independent speaker identification in birds, bandpass filtering and spectral subtraction still left daytime wind and traffic recordings at 68% identification accuracy, against 95–100% in quiet conditions. The shipping band in Comparison of Two Soundscapes overlaps target signals too broadly to subtract. Deployed models are also usually trained on raw audio, so denoising only at unlearning time would mismatch the training distribution. We include an enhancement-then-PGU arm anyway. If it closes the shortcut gap, the simpler pipeline wins and we will say so.
Why not just augment the forget set with random noise and run ordinary gradient ascent? This is the most serious threat, because re-mixing is half of our method. The ablation that uses re-mixed forget clips with the plain PGU subspace (and a further arm with plain gradient ascent) answers it directly. Our hypothesis is that re-mixing alone fixes cross-environment forgetting but not same-scene collateral damage. The update can still erode background features shared with retained speakers, and the background-only protected subspace is what prevents that. If the ablation matches the full method on same-scene retained accuracy, the contribution reduces to an evaluation finding plus a data trick, and we will report it that way.
Why not just use PGU, since its retained set already contains same-scene speakers? If the retained data covers the home scenes, PGU's protected subspace may already contain the background directions. We test this by varying how many retained speakers share each forget speaker's scene, from none to many. Our prediction is that PGU matches our method when coverage is high and degrades when the forget speaker's environment is rare in the retained data. That is the realistic case for a private home recording. If PGU is flat across coverage levels, the background-only subspace adds nothing.
Proposed Experiments
The first experiment is the diagnostic. On the confounded LibriSpeech/VCTK benchmark, we measure the environment shortcut gap for every baseline, sweeping signal-to-noise ratio from clean to heavily masked. The prediction is that the gap is near zero for retraining and grows as the noise gets louder for gradient ascent and Boundary Unlearning. Finding no gap anywhere would falsify the central premise. In that case the remaining contribution is the benchmark itself.
The second experiment compares methods at matched compute. We report:
- forget accuracy in-environment and cross-environment;
- retained accuracy, split into same-scene and other-scene speakers;
- speaker-level membership AUC on foreign-environment queries;
- forget accuracy after the relearning attack;
- linear-probe decodability of identity from intermediate layers;
- wall-clock time relative to retraining.
Because the speaker-level attack reaches AUC 0.9999–1.0000 on undefended self-supervised models, the meaningful target is closeness to the retrained model's AUC, not to 0.5.
The third experiment uses PGU-style coverage to settle the PGU question. It varies retained-speaker overlap with forget scenes as described above.
The fourth experiment transfers the procedure to VoxCeleb1 with natural confounding. There we test whether low-signal-to-noise forget speakers show larger post-unlearning leakage than high-signal-to-noise ones under each method.
The decisive check comes from combining the first two experiments. If standard methods forget in-environment but not cross-environment, and our method closes that gap without the same-scene collapse that gradient ascent shows, then the field's current forgetting metric for audio is measuring the room rather than the voice, and there is a cheap correction for it.
References
- Speech Unlearning
- Unlearning LLM-Based Speech Recognition Models
- Spatio-Temporal Attention Pooling for Audio Scene Classification
- Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection
- Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning
- Comparison of Two Soundscapes: An Opportunity to Assess the Dominance of Biophony Versus Anthropophony
- Membership Inference Attacks Against Self-supervised Speech Models
- Boundary Unlearning: Rapid Forgetting of Deep Networks via Shifting the Decision Boundary
- Text-independent speaker identification in birds