Speaker-Level Certified Forgetting for Emotion Recognition with a Public Low-Dimensional Subspace

People ask to have their voice data deleted from emotion and mental-health models for reasons that are rarely random. A user sees the system label them as angry, or a clinic patient notices a depression score they did not expect, and only then requests removal. Machine unlearning (removing a person's influence from a trained model) has mostly assumed the opposite: that deletion requests arrive independently of what the model has shown. Forget Unlearning proves that once requests depend on past model outputs, even retraining from scratch can leak the deleted person's data. Its remedy is differential privacy (DP), a guarantee that the model's outputs barely change whether or not any single record is present. The paper shows DP for the remaining records is a necessary condition. Emotional speech is the setting where that remedy is hardest to apply. The corpora are tiny: CREMA-D has 91 speakers and IEMOCAP has 10. The unit a person wants deleted is their whole voice, dozens of utterances, not one clip. DP noise grows with model dimension and shrinks with the number of protected people, so a 768-dimensional emotion classifier protected at the speaker level over 70 or so training speakers may be close to useless.

We propose to make adaptive-safe speaker deletion workable at this scale. The privacy-sensitive part of the model is confined to a very small number of directions estimated only from public emotional and neutral speech. The pipeline has three parts. A frozen self-supervised speech encoder is pretrained on public audio and never touched by private users. A fixed projection, learned from public acted-emotion corpora and whitened with neutral-speech statistics, maps its features to a handful of emotion-relevant dimensions. A small strongly convex classifier is trained on the private users with noisy gradient steps and per-speaker sensitivity control. Deleting a speaker means a few further noisy steps on the reduced dataset, which is covered by existing certified-unlearning analyses. The bet is that the public subspace, not the deletion algorithm, is what makes the privacy cost affordable when emotional data are scarce.

Core Claim

Under speaker-level deletion privacy that holds against adaptive requests, an emotion or depression classifier restricted to a public $k$-dimensional subspace, with $k$ on the order of tens, will retain substantially more accuracy than the same certified procedure run on full encoder features. It will also beat a model trained on public data alone, which is the bar any private model must clear to justify touching user data. The expected mechanism comes from standard private convex optimization. Excess risk grows roughly with $\sqrt{k}$ and shrinks with the number of protected speakers times the privacy parameter $\varepsilon$. Cutting $k$ from 768 to around 16 is therefore the main lever available when the speaker count cannot grow.

This differs from the closest work in specific ways. Towards Machine Unlearning for Paralinguistic Speech Processing is the only prior unlearning study on emotion (CREMA-D) and depression (E-DAIC) speech. It uses SISA++, which splits the data into disjoint shards, trains a model per shard, averages their weights, and retrains only the affected shard. Its deletions are random users, it gives no privacy guarantee or membership test, and it splits 91 speakers across up to 8 shards. We use no sharding, model adaptive deletions, and certify the result.

Machine Unlearning in Low-Dimensional Feature Subspace (LOFT) also forgets through a small projection over a frozen backbone. However, it learns that projection from the private retain and forget data and states that it has no DP guarantee. In our design the projection is a function of public data only, so it leaks nothing about private users. All forgetting happens in the noisy convex head.

Certified Machine Unlearning via Noisy Stochastic Gradient Descent supplies the deletion mechanism, projected noisy stochastic gradient descent (PNSGD), along with its sequential accounting. It was demonstrated on binary vision tasks with per-sample deletion over fixed pretrained features. We change the protected unit to the speaker and move to a regime where the number of protected units is two to three orders of magnitude smaller.

Research Program

The first component is the public subspace. We take a frozen wav2vec 2.0 or WavLM encoder and use the trainable layer-weighted average of middle layers that Emotion Recognition from Speech Using wav2vec 2.0 Embeddings found most emotion-bearing. That paper also found pretrained-only features beat ASR-fine-tuned ones, which supports keeping the encoder frozen. The layer weights are fit on public data only.

Per-speaker normalization helps a lot in that paper: IEMOCAP average recall rises from 65.8 to 67.2 and RAVDESS from 75.7 to 84.3. But it needs several utterances per speaker, and the statistics would themselves be private. We replace it with whitening by statistics from public neutral speech (TIMIT). This follows the neutral-anchoring idea of Using neutral speech models for emotional speech analysis, which models emotion as deviation from a neutral reference.

On top of the whitened features we fit a $k$-dimensional discriminative projection from public labeled emotional corpora (RAVDESS, EMO-DB, and IEMOCAP when it is not the target). We use a PLDA-style emotion subspace, in the spirit of Emotion clustering based on probabilistic linear discriminant analysis, whose emotion subspaces transferred across corpora. We vary $k$ from 4 to 128.

The second component is the private head and its deletion mechanism. The head is an L2-regularized multinomial logistic regression for CREMA-D emotion, or a ridge regressor for E-DAIC depression scores. Parameters are projected onto a ball so the loss is strongly convex and smooth. Each step computes one summed gradient per speaker, clipped to a fixed norm, so a single speaker's total influence is bounded at the speaker level rather than through a group-privacy blow-up over that speaker's utterance count. Gaussian noise is then added.

Training follows Noisy-GD from Forget Unlearning, which is the analysis that covers adaptive request sequences. PNSGD's Wasserstein-distance accounting is used as the more efficient variant for long request sequences. A deletion removes the speaker and runs a small number of further noisy steps from the current parameters.

Because the encoder and subspace are independent of private users, the whole released pipeline is post-processing of the private head. The adaptive-deletion guarantee then reduces to the convex case those papers prove. The analytical deliverable is an explicit $(q,\varepsilon)$ deletion-privacy accounting at the speaker level for each operating point we report.

The third component is an adaptive attack harness. Following the threat model of Adaptive Machine Unlearning, an adversary observes each released model and requests deletion of the speakers on whom the model is most confident or most wrong. These are exactly the users most likely to react in practice. The adversary then compares the models before and after deletion. We measure membership-inference advantage on deleted speakers against never-seen speakers. On E-DAIC we also measure inference of the deleted participant's depression score from the parameter change.

Baselines

The naive reference point is a model trained only on the public corpora and evaluated on the private test speakers. It uses no private data, so it needs no deletion at all. This is the baseline that most threatens the proposal. If the certified private head cannot beat it, private training is not worth doing. We will run it with the identical frozen encoder, whitening, subspace and head, and the same hyperparameter search budget. The win criterion is a statistically significant gain in unweighted average recall on CREMA-D across speaker-disjoint folds, and lower MAE on E-DAIC, at $\varepsilon \le 1$.

The second baseline is the non-private oracle: the same head retrained from scratch without noise after each deletion. It sets the utility ceiling, and the attack harness is expected to show its adaptive leakage. For context, the closest reported frozen-feature numbers are 65.8 average recall on IEMOCAP and 75.7 on RAVDESS with global normalization (Emotion Recognition from Speech Using wav2vec 2.0 Embeddings). There is no reported result found on Lacuna for frozen-feature emotion recognition accuracy on CREMA-D, so we will measure it ourselves.

The strongest existing paralinguistic unlearning method is SISA++ (Towards Machine Unlearning for Paralinguistic Speech Processing), using its reported best recipe, TRILLsson features with a Transformer head, with 4 and 8 shards. No reported result found on Lacuna for SISA++ accuracy or F1 numbers on CREMA-D or E-DAIC, so we will rerun it on our splits. Beating it means matching or exceeding its utility while reducing adaptive membership advantage to within confidence of chance, where SISA++ is expected to sit well above chance.

The strongest certified competitor is the same Noisy-GD/PNSGD head trained on the full 768- or 1024-dimensional whitened features with identical speaker-level clipping, noise calibration, $\varepsilon$, step count and tuning budget. This is the PNSGD recipe of Certified Machine Unlearning via Noisy Stochastic Gradient Descent moved to speech. That paper reports roughly 98% accuracy on CIFAR-10 cat-vs-ship at $\varepsilon = 0.01$, but that is a binary task with tens of thousands of protected samples, not tens of protected speakers. This comparison is also the ablation that isolates our key ingredient. A second ablation learns the $k$-dimensional subspace from private data, LOFT-style, and accounts for it with DP. This separates the benefit of low dimension from the benefit of the subspace costing no privacy. Finally, heuristic gradient-ascent unlearning, as in Speech Unlearning, serves as the uncertified reference.

Why not just…?

Why not just retrain from scratch, since emotion datasets are tiny? On a small head, retraining takes seconds, so compute is not our argument. The objection is privacy. Forget Unlearning proves that noiseless retraining leaks under adaptive requests, and our attack harness will measure that leak directly on CREMA-D and E-DAIC. If the measured adaptive advantage of noiseless retraining is indistinguishable from chance on these corpora, we will report that the theoretical threat is empirically weak here. DP retraining from scratch, as opposed to fine-tuning, is a legitimate equivalent to our deletion step. We include it, and expect it to share our accuracy, because the subspace drives the gain.

Why not just train on public emotional corpora and never touch user data? This is the real threat. Cross-corpus emotion transfer is weak, as Affect Models Have Weak Generalizability to Atypical Speech documents for atypical speech. That weakness suggests private data should help, especially for clinical E-DAIC, where no public depression corpus exists in the file. Still, the public-only baseline is our primary bar. If it wins at practical $\varepsilon$, the honest conclusion is that certified private personalization is not yet worth it for scarce affective data. That is a publishable negative result.

Why not just use SISA++ with more shards or a private aggregator? Sharding divides speakers that are already scarce. On IEMOCAP, 8 shards leaves barely one speaker per shard. Private SISA (Adaptive Machine Unlearning) fixes the adaptivity but its privacy budget runs out with queries, forcing full retraining. Our comparison against SISA++ at matched speaker splits will show whether sharding or noise costs more accuracy at this scale.

Why not just add DP noise to the full features? That is exactly the full-dimension certified baseline. If it matches the subspace head, our central claim fails.

Proposed Experiments

The main experiment uses CREMA-D (six emotions, speaker-disjoint 5-fold) and E-DAIC (depression regression on 5-second chunks, participant-level deletion). It traces utility against speaker-level $\varepsilon$ in $\{0.5, 1, 2, 4, 8\}$ for each $k$, under sequences of 1 to 20 adaptive deletions.

The first decisive check is whether some $k$ well below the full dimension gives a significantly better utility–privacy curve than full features. It also asks whether that curve crosses above the public-only baseline at $\varepsilon \le 1$. Our hypothesis predicts a U-shape in $k$: too few dimensions lose valence information, which acoustic models already struggle with, and too many pay the noise cost.

The second check is the attack harness. Membership and score-inference advantage against our head should stay within the certified bound and near chance. Against noiseless retraining and SISA++, it should be measurably above chance; Speech Unlearning reports membership-attack scores often around 40–50% for heuristic methods.

The third check tests robustness to the public–private gap. We repeat the experiment using IEMOCAP as private data and only acted corpora as public. This asks whether a subspace learned from acted speech still serves semi-spontaneous dyadic speech.