Erasing Voice, Keeping Accent: Selective Speaker Subspaces in Frozen Speech Models

Speech models trained without labels, such as HuBERT, wav2vec 2.0 and WavLM, learn internal features that carry both what was said and who said it. Systems that convert one voice into another, or that anonymize a speaker, need the first without the second. Today's fixes take one of three forms. The first is to re-train the whole model to forget the speaker. The second is to round its features into a small set of discrete codes, which leaks less voice but also makes speech mumbled and flat. The third is to discard everything except phoneme probabilities from a supervised recognizer. All three throw away too much. In particular they erase the speaker's accent and pitch contour along with their timbre, so a Scottish speaker converted to a new voice comes out sounding generically standard, or picks up the accent of the training language.

This note proposes a training-free alternative. Take one intermediate layer of a frozen model and remove a small, linear set of directions: the directions along which speakers differ from other speakers who share their accent. Directions that encode accent and pitch are explicitly protected. In two sentences: speaker identity and phonetic content are already known to sit in nearly orthogonal directions of these models, so speaker identity can be projected out with a matrix rather than re-trained away. We go one step further and ask whether "speaker" can be split into timbre, which we delete, and accent plus prosody, which we keep. Either answer is informative. Success yields a cheap content representation that preserves accent. Failure shows that accent and timbre are entangled in the same directions, which would explain the "accent trap" reported in anonymization work.

What is known and what is missing

Three findings motivate the proposal. First, layer-wise analyses show that information is stratified by depth. Comparative layer-wise analysis of self-supervised speech models traced phonetic content with canonical correlation analysis and found that it peaks in middle layers of contrastive models but late layers of HuBERT and WavLM. It also found that the single best layer beats the standard learned weighted sum of all layers by 4–9% relative on recognition. Second, geometry analyses show that the stratification is clean. Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations measured, across six models, how far the phone-centroid and speaker-centroid subspaces overlap. They are nearly orthogonal, and orthogonality correlates with phone probing accuracy (ρ = 0.54). That analysis was purely diagnostic, on LibriSpeech dev-clean. Third, Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces turned this geometry into an operation. Its "subspace collapsing" projects features onto the complement of the speaker subspace, and it improved phone discrimination measured by the ABX test (whether a sound is closer to a same-phone than a different-phone example). That work used only small predictive-coding models, a single layer, no generative task, and a speaker subspace that lumps every between-speaker difference together.

That last point is the gap this note targets. Improving Voice Quality in Speech Anonymization With Just Perception-Informed Losses reports the accent trap: content bottlenecks map foreign phones into the categories of the training language. ContentVec re-trains HuBERT with voice-converted teachers, a contrastive loss over pitch- and formant-shifted views, and speaker conditioning. It reaches strong disentanglement (speaker identification accuracy drops from 73.7% to 37.7%) but notes that accent is suppressed alongside identity. Its training is also a full pre-training run with sensitive hyperparameters. The supervised-recognizer route in Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance constrains our design. Sweeping an 18-layer Conformer, it found a direct tension: deeper layers lower character error but lower pitch correlation. It needed 10,000 transcribed hours to find its middle-layer compromise. A content representation that is good at both content and pitch has to come from somewhere other than depth alone.

Core Claim

At the phonetic-peak layer of a frozen self-supervised model, the between-speaker variance that remains after conditioning on accent is low-rank and nearly orthogonal both to phone identity and to the directions that predict accent and fundamental frequency (F0, the pitch of the voice). Projecting out only this "within-accent speaker" subspace can therefore match a full speaker-subspace projection, and approach ContentVec, at removing voice identity, while keeping substantially more accent and pitch information than either. It needs speaker and accent metadata for a small fitting set, but no transcripts and no re-training.

Concretely, let $h_t \in \mathbb{R}^d$ be the layer-$\ell$ feature at frame $t$, and define three covariance matrices over the fitting set:

  • $S_{s|a}$, the covariance of speaker means around their accent group's mean;
  • $S_a$, the covariance of accent-group means;
  • $S_{f}$, the covariance explained by a linear regression of frame-level log-F0 on $h_t$.

We find the speaker directions $u$ that are as uninformative as possible about the protected attributes by solving

$$S_{s|a}\, u = \lambda\, (S_a + S_f + \epsilon I)\, u ,$$

where $\epsilon$ is a small ridge term. We keep the top $k$ generalized eigenvectors, orthonormalize them into $U \in \mathbb{R}^{d\times k}$, and apply

$$\tilde h_t = (I - U U^\top)\, h_t .$$

Setting $S_a = S_f = 0$ recovers ordinary total-speaker subspace collapsing. That reduction is the ablation that isolates the proposal's one new ingredient. The rank $k$ plays the role of bottleneck width, and it is swept rather than tuned by hand.

Research Program

The work has three stages that build on each other.

Locating the right layer. For HuBERT Base, WavLM Base and wav2vec 2.0 Base, we fit the projection at every layer. We measure linear and two-layer-MLP probe accuracy for phones, speakers, accents and F0, before and after projection. The MLP probes matter because a linear projection can hide speaker information from a linear reader while leaving it recoverable non-linearly. We expect the best layer to differ by architecture, following the contrastive-versus-predictive split reported for layer-wise canonical correlation. We report both the probe-selected layer and the layer chosen by the cheaper geometry criterion of the orthogonality paper, to see whether layer selection can be done without probes.

Testing selectivity. Accent labels come from VCTK's 109 speakers with diverse accents. Fitting uses held-out VCTK speakers plus LibriSpeech train-clean-100 speakers (speaker IDs only). Evaluation is on unseen VCTK speakers. We trace three curves as the rank $k$ grows: speaker erasure, accent retention and F0 retention. The comparison is the selective projection against total-speaker collapsing at matched speaker erasure.

Downstream use. The projected features feed the S3PRL-VC recognition-synthesis pipeline for zero-shot any-to-any conversion (S3PRL-VC, A Comparative Study of Self-Supervised Speech Representation Based Voice Conversion). The synthesizer is trained on VCTK with a VoxCeleb-pretrained speaker encoder, exactly as in that study, so only the content features change. As a stretch test of the accent trap itself, we run German-to-English anonymization on mlsGerman with the setup of the perception-informed-loss paper.

Baselines

The naive reference point is the raw frozen feature at the same layer, plus the SUPERB-style learned weighted sum of all layers (SUPERB). Both are fed into the identical VC synthesizer. The comparative S3PRL-VC study already shows they leak source timbre in any-to-any conversion, so beating them on speaker similarity is necessary but not sufficient.

The strongest representation-level competitor is ContentVec, run from the released checkpoint (auspicious3000/contentvec) at its recommended output layer. It is the strongest because it is the best reported self-supervised disentangler. In its own paper it reports speaker identification accuracy of 37.7%, across-speaker ABX of 6.32 against 7.20 for HuBERT, and phone error 0.049 against 0.054. We cannot match its compute, and the point is not to. For fairness we compare at matched speaker-identification accuracy: we choose $k$ so that our SID equals 37.7% on SUPERB's VoxCeleb1 protocol, then compare ABX, phone error, accent retention and F0 retention. Beating it means equal or better ABX, together with clearly higher accent-probe accuracy on unseen VCTK speakers, at that matched SID. Losing on ABX by a small margin while winning on accent would still be a useful trade-off for a training-free method. Losing on accent would falsify the core claim.

The strongest bottleneck competitor is k-means and product quantization on the same layer, following the comparative VC study, which found product quantization to be the better-balanced discretization. Codebook sizes are swept with the same tuning budget as our rank sweep.

At the system level, the reference is the auxiliary-loss zero-shot converter of Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features. It reports VCTK CER of 6.15% and MOS of 3.66 against 3.24 for FragmentVC. FragmentVC reports 92.5% unseen-to-unseen speaker verification accuracy, or 86.5% with a single target utterance. Those numbers come from different synthesizers, so they serve as context only. The fair test is our projection inside the shared S3PRL-VC synthesizer. For the S3PRL-VC numbers themselves on VCC2020, no reported result was found on Lacuna, so we will re-run them. For accent preservation under conversion, no reported result was found on Lacuna for any method, so this proposal establishes the first.

The key ablation is total-speaker subspace collapsing ($S_a = S_f = 0$) at the same layer and the same speaker erasure. If it preserves accent and pitch as well as the selective version, the selective machinery is unnecessary.

Why not just…?

Why not just subtract each utterance's mean feature? This is the most serious simple threat, because speaker centroids are close to utterance means, and mean subtraction needs no labels at all. But accent is also a property that is constant within an utterance, so mean subtraction should remove accent centroids along with speaker centroids. It also does nothing about time-varying timbre cues. We include it as a baseline at the same layer. If it matches the selective projection on accent retention, our claim is wrong and we will say so.

Why not just use ContentVec? ContentVec's own authors note that it suppresses accent and loses slight content, and it requires full pre-training with sensitive tuning. The matched-SID comparison above directly measures whether a frozen-model projection can match its erasure while keeping accent. If ContentVec also retains accent at matched erasure, the selective claim loses its main motivation.

Why not just pick the best layer and quantize it? The comparative S3PRL-VC study shows that post-hoc discretization raises word error. The streaming Conformer study shows that discrete or deeper bottlenecks cost pitch correlation. Quantization has no way to choose which between-speaker variation to drop. The product-quantization baseline at the selected layer, compared at matched speaker verification acceptance, settles this.

Why not just use supervised phonetic posteriorgrams? Posteriorgrams require transcribed data, and by construction they discard continuous pitch. The streaming study measured lower naturalness for its posteriorgram-style baseline than for its intermediate bottleneck features (MOS 3.71 vs. 3.85). Mapping onto a fixed phone inventory is also the mechanism behind the accent trap. Posteriorgrams therefore answer a different question than accent-preserving erasure.

Proposed Experiments and Decisive Checks

On LibriSpeech and Libri-light we report the following for each model, layer and rank: across-speaker ABX, SUPERB phone recognition, and SUPERB speaker identification and verification, with linear and MLP speaker probes. On unseen VCTK speakers we report accent-probe accuracy and F0 regression correlation before and after projection. For any-to-any VCTK conversion we report:

  • speaker verification acceptance against the target;
  • recognizer CER on converted speech;
  • pitch Pearson correlation with the source;
  • accent-classifier agreement between source and converted speech;
  • a listening test of naturalness and of accent and speaker similarity.

The decisive check is a single plot: accent retention against speaker erasure for the selective projection, total-speaker collapsing, mean subtraction, product quantization and ContentVec. The proposal succeeds if the selective curve dominates the others at matched erasure in both probing and conversion. The proposal fails informatively if every curve coincides. That would mean accent and timbre share directions in these models, and the accent trap cannot be fixed by any linear content bottleneck. A second check guards against false success: if MLP probes recover speakers well after projection, or converted speech still verifies as the source speaker, the linear erasure is cosmetic and the conversion results take precedence over the probes.