Removing Speaker Identity by Timescale in Self-Supervised Speech Features

Speech emotion recognizers trained on a few hundred labeled clips tend to learn who is speaking instead of how they feel. When the test speakers were never seen in training, accuracy drops sharply. Current fixes each carry a cost. Adversarial training makes the network fight a speaker classifier, which is expensive and unstable. Per-speaker normalization needs several recordings of each test speaker. Speaker-embedding conditioning brings in yet another large model. This proposal starts from a simple observation about signals. A speaker's voice is roughly constant across an utterance, while emotional prosody (rises and falls in pitch, energy, and rhythm) changes over syllables and phrases. If that holds inside the features of a pretrained speech model, then speaker identity should sit mostly in the slowest-changing part of each feature's trajectory over time. It can then be removed there, and only there, from a single utterance, without any emotion labels.

Concretely, we pass frozen features from a self-supervised speech model through a fixed bank of temporal filters of different lengths. A self-supervised model is one pretrained on unlabeled audio, such as HuBERT or WavLM. The filter bank is a multi-scale temporal convolution: filters of different lengths applied along time, so that each one isolates a band of slow or fast variation. From speaker-labeled but emotion-unlabeled speech, we estimate a separate "speaker direction" subspace for each band. We then project that subspace out of the features band by band, before training a small emotion classifier on 1–10% of the emotion labels. The contribution is twofold. First, a measurement of where speaker and emotion information live across timescales in these features, which nobody has reported. Second, a cheap, non-adversarial, single-utterance normalizer whose value depends on that measurement being right.

Core Claim

In frozen HuBERT or WavLM features, speaker identity is concentrated in the utterance-level mean and the slowest modulation band. Most emotion information, particularly for categorical emotion and valence, is carried in faster bands. A small but important part of arousal sits in the slow band. Projecting out a speaker subspace estimated separately per band therefore removes more speaker information per unit of emotion information lost than projecting the same subspace out of the undecomposed features. It also loses less than simply subtracting each utterance's mean. On speaker-independent splits with scarce labels, this should close most of the gap to oracle multi-utterance speaker normalization. It should also match adversarial speaker normalization at a small fraction of its training cost.

If the first part fails, the method has no reason to work, and we will report it as a negative finding. That failure would mean speaker information is spread evenly across bands.

How This Relates to Prior Work

The closest work removes speaker information without regard to timescale. Speaker Normalization for Self-supervised Speech Emotion Recognition attaches an adversarial speaker-identification head to HuBERT and negates its gradients. It uses VoxCeleb as a non-overlapping speaker corpus, and it reaches its best speaker-independent IEMOCAP result only when all upstream parameters are fine-tuned. Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition applies the same gradient-reversal idea to handcrafted features. Speaker-Aware Multi-Task Learning for Speech Emotion Recognition instead pushes speaker and emotion embeddings apart with a cosine penalty. All three treat the whole feature sequence as one object to be made speaker-invariant. We replace their minimax training with a closed-form projection, and we apply it separately per timescale.

Factor Analysis Based Speaker Normalisation for Continuous Emotion Prediction supplies the projection idea: split features into speaker and emotion subspaces. Classification of cognitive load from speech using an i-vector framework similarly factors out speaker traits. We reuse this family of methods on self-supervised features and add the temporal decomposition.

Two papers motivate the problem and constrain the design. Emotion Recognition from Speech Using wav2vec 2.0 Embeddings shows that per-speaker normalization of wav2vec 2.0 features improves IEMOCAP recall from 65.8% to 67.2%, and RAVDESS recall from 75.7% to 84.3%. It also shows that this requires many utterances per test speaker, which is exactly the constraint we remove. Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognition shows that speaker variation splits into feature shift, which mostly hurts arousal, and label shift, which mostly hurts valence. It also shows that proxy-speaker calibration fails for outlier speakers. Our method targets only feature shift, and we say so. Because calibration can be stacked on top, label shift is left to that method rather than claimed.

The multi-scale machinery is borrowed from three sources. TSception sets temporal kernel lengths as fixed ratios of the sampling rate to target known frequency bands. We do the same relative to the 50 Hz frame rate of self-supervised speech features. NeXt-TDNN uses parallel depthwise temporal convolutions with kernel sizes from 7 to 65 frames for speaker verification. That is evidence that speaker cues are extracted across a range of scales, and it is a reason we must measure rather than assume. BioFormer normalizes cross-subject variation in EEG and ECG by realigning individual frequency bands. It reports that the approach weakens on irregular, event-driven signals. Speech prosody is such a signal, so we deliberately avoid BioFormer's phase-alignment machinery and use a simpler additive decomposition.

Two further works leave room for this one. Multi-Scale Temporal Transformer For Speech Emotion Recognition shows that multi-scale pooling over frozen HuBERT-large features improves emotion accuracy, but it never examines speaker leakage. GM-TCNet builds multi-scale dilated convolutions on MFCC features, the standard hand-engineered spectral features. Neither uses timescale to separate speaker from emotion.

Research Program

The decomposition is a temporal Laplacian pyramid, meaning the features are split into a stack of bands that sum back to the original. Let $x_t$ be the frame-$t$ feature vector from one layer of a frozen encoder. Let $A_s$ be a centered moving average of length $s$ frames, with lengths $s_1 < s_2 < \dots < s_K$ and $A_{s_{K+1}}$ defined as the utterance mean. Band $b$ is

$$x^{(b)}_t = (A_{s_b} x)_t - (A_{s_{b+1}} x)_t,$$

so the bands sum back exactly to the lightly smoothed input. The top band is the utterance-constant component. We use lengths near 5, 25 and 100 frames, roughly 0.1 s, 0.5 s and 2 s. These separate phone-level, syllable-level and phrase-level variation. Because the filters are fixed, they add no parameters and no training.

For each band, we estimate the between-speaker scatter of band-averaged features using speaker IDs only. These come from the unlabeled training audio of the target corpus, and optionally from VoxCeleb as in Gat et al. We take the top $r_b$ eigenvectors $U_b$ and project them out:

$$\tilde{x}^{(b)} = (I - U_b U_b^\top)\, x^{(b)}.$$

Here $r_b$ is the per-band rank, chosen on a speaker-disjoint development split. The normalized bands are recombined, or kept as separate channels. They are then fed to a small emotion head: attentive pooling followed by a linear layer. The head is trained on 1%, 10% or 100% of emotion labels. The normalizer needs one utterance at test time and no emotion labels at any point.

The program has three stages. The first is a measurement study. For each encoder layer and band, we train linear speaker-ID probes and emotion probes, then plot information by band. We also measure how much arousal the top band carries. This stage answers whether the premise is true. The second stage builds the normalizer and chooses ranks. The third stage runs the low-label, unseen-speaker evaluation, the ablations, and residual speaker-leakage audits.

Baselines

The naive reference point is a frozen encoder with a learned weighted sum of layers, global feature normalization, attentive pooling and a linear head, with no speaker treatment. This mirrors the global-normalization setting of Emotion Recognition from Speech Using wav2vec 2.0 Embeddings, which reports 65.8% recall on IEMOCAP. The same paper's per-speaker normalization, reported at 67.2% recall, serves as an oracle. It sees many utterances of every test speaker, which our method does not. If we recover most of the gap between these two, the single-utterance claim holds.

The strongest existing method is the adversarial speaker normalization of Speaker Normalization for Self-supervised Speech Emotion Recognition. It reports 74.2% weighted accuracy on speaker-independent 5-fold IEMOCAP with HuBERT Large and full fine-tuning. We run both of its configurations. The first is the projector variant on a frozen encoder, the fair like-for-like comparison. The second is full fine-tuning, reported as a compute-heavy upper reference. Both use our encoder, our label fractions, and our splits, the same speaker data for the adversary as for our subspace estimation, and the same hyperparameter-search budget. We define beating it as matching or exceeding the frozen-encoder adversarial variant on unweighted accuracy at 10% labels, while using at least an order of magnitude less training compute. Matching full fine-tuning is not required for the claim.

We also re-run Speaker-Aware Multi-Task Learning for Speech Emotion Recognition. It reports gains of 1.16% unweighted accuracy over its baseline in the speaker-independent setting, and we test it under the same protocol. On IEMOCAP with frozen HuBERT-large, Multi-Scale Temporal Transformer For Speech Emotion Recognition reports 70.60% weighted accuracy. It is included to show whether multi-scale modeling helps by itself, with no speaker treatment.

On MSP-Podcast, the reference is PAPT from Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognition. It reports that arousal concordance averaged over unseen speakers rose from 0.263 to 0.297. We compare against it directly and stack our normalizer with its label calibration.

We apply the reality-check protocol of Designing and Evaluating Speech Emotion Recognition Systems throughout: strictly speaker-independent splits and the standard four-class set including happy. Reported numbers from other protocols are context only. Every comparison uses numbers we re-run ourselves.

Three ablations isolate the key ingredient. The first projects the same total rank out of the undecomposed features, removing the per-band separation. The second only subtracts each utterance's mean, which is instance normalization. The third replaces the fixed filters with learned depthwise multi-scale convolutions in the style of NeXt-TDNN, followed by a per-band adversary instead of a projection.

Why not just…?

Why not just subtract each utterance's mean (instance normalization)? This is the most serious threat. If speaker identity lives entirely in the utterance-constant component, mean subtraction captures most of our benefit for free. Our hypothesis predicts that it will also delete arousal level (overall loudness and pitch height), which Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognition identifies as the dimension most affected by feature shift. It should therefore underperform specifically on arousal and on high-arousal categories. The instance-normalization ablation tests exactly this. If it ties our method on arousal concordance and on unweighted accuracy, we will report that the simple method suffices, and the contribution reduces to the measurement study.

Why not just use adversarial speaker removal as Gat et al. do? It works, but its best result needs full fine-tuning of a large encoder under unstable minimax training. It also cannot say which part of the signal it removed. Our projection is closed-form, needs no emotion labels to fit, and can be inspected band by band. The head-to-head comparison with the frozen-encoder adversarial variant, at matched labels and speaker data, settles whether the simplicity costs accuracy.

Why not just condition on an off-the-shelf speaker embedding? Speaker-Aware Multi-Task Learning for Speech Emotion Recognition reports beating an external-speaker-embedding baseline. Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognition finds that injecting speaker embeddings alone does not close the unseen-speaker gap. Conditioning also lets a low-label classifier use identity as a shortcut rather than removing it. We include ECAPA-style conditioning at matched label fractions, and we measure residual speaker leakage as well as accuracy.

Why not just use a better multi-scale head without any speaker treatment? Multi-Scale Temporal Transformer For Speech Emotion Recognition shows multi-scale structure helps, so some of our gain could simply come from a better head. The multi-scale-head-only baseline and the undecomposed-projection ablation together separate the two effects. Our claim requires the band-wise projection to beat both.

Proposed Experiments and Decisive Checks

We use four datasets. IEMOCAP is evaluated under leave-one-session-out, four classes, with unweighted and weighted accuracy. CREMA-D has 91 actors, which gives more stable speaker-independent estimates. MSP-Podcast supplies arousal and valence concordance, both overall and averaged per unseen speaker. RAVDESS allows comparison with the per-speaker-normalization oracle. Label fractions are 1%, 10% and 100%, sampled by speaker to avoid leakage, with five seeds each. We use HuBERT-large and WavLM as frozen encoders.

The first decisive check comes from the measurement study. Speaker-probe accuracy must fall steeply from the slowest band to the fastest, while emotion-probe accuracy in the fast bands stays well above chance. If speaker information is flat across bands, the method is unmotivated, and we report that finding.

The second check is the main result. At 10% labels on speaker-independent splits, band-wise projection must beat undecomposed projection and instance normalization on unweighted accuracy and on arousal concordance. It must also recover most of the gap between global normalization and the oracle per-speaker normalization.

The third check is a leakage audit. After normalization, a freshly trained speaker probe on pooled features should drop toward chance. The retained emotion information should exceed what the adversarial baseline retains at equal speaker leakage. We will plot speaker-probe accuracy against emotion accuracy as a tradeoff curve for every method, since a single operating point can hide a worse tradeoff.

We will analyze valence separately. Valence is driven by label shift, so we expect little gain there and will not claim any.