Aligning Speakers Like Knowledge Graphs: Phonotactic Similarity Flooding for Speaker-Invariant Speech Units

Two people saying the same sentence produce very different sound waves. Their vocal tracts differ in length, their pitch differs, and their vowels sit in different places in acoustic space. What they share is the order in which sounds follow one another, because both speak the same language. This proposal treats that shared ordering as a fingerprint. We split each speaker's speech into their own private set of sound clusters and draw, for each speaker, a graph of which clusters tend to follow which. We then find which cluster of speaker A corresponds to which cluster of speaker B by matching the two graphs. The matching uses the rule "two clusters correspond if the clusters that precede and follow them correspond." The units that come out of the matching are the content of speech. The per-speaker table that maps each shared unit to that speaker's acoustic realization is the speaker's identity.

The opportunity comes from a result in a distant field. In ontology matching, GMO aligned ontologies with an F-measure of 0.95 even after every class and property name was replaced by a random string. Graph structure alone carried the correspondence. In speech, a speaker's acoustic idiosyncrasies play the role of those scrambled labels. Current voice conversion and unit discovery systems remove identity from content by squeezing representations through a narrow channel (an "information bottleneck") or by relying on large pretrained encoders. Both trade away naturalness or assume resources that low-resource languages lack. Recovering content as the structure shared across speakers offers a different route, and that route can be tested directly.

Core Claim

Our claim is that per-speaker acoustic clustering followed by cross-speaker graph alignment with similarity flooding yields more speaker-invariant units than clustering all speakers together. The gain should hold even when the global clustering is given speaker normalization. It should grow with how strongly and how non-linearly speakers differ, as with cross-gender, child versus adult, or synthetically warped voices.

The reason is that normalization methods can only undo speaker differences that are close to a shift or rescaling of the features. The transition structure between sound categories does not change under any smooth acoustic warp. If the claim is false, a per-speaker mean subtraction followed by global k-means will match our units in a controlled warp test. That outcome would settle that structure adds nothing.

Relation to Prior Work

The graph side supplies the method. Similarity flooding computes pairwise similarities iteratively until they stop changing. What Makes Entities Similar? showed that common neural entity-alignment models are, in effect, solving for this same fixed point, and that solving it directly works well. Its TransFlood method improved Hits@1 over MTransE by more than 10% on DBP15K French-English. Its self-propagation term keeps each node's own evidence from being washed out by its neighbours, and we reuse it for the same purpose.

PipEA contributes three pieces we also reuse:

  • propagation from very few seed matches;
  • a matched-neighbourhood consistency check;
  • the Sinkhorn operator for near one-to-one matching.

PipEA's guarantee assumes the subgraphs being matched have identical structure, and speech violates that assumption. Speakers drop or merge sounds, and one speaker's single vowel cluster may be split into two allophones by another.

AliNet handles exactly this kind of mismatch in knowledge graphs. It uses gated two-hop neighbourhoods, reaching 0.552 Hits@1 on DBP15K FR-EN. We adopt its idea by adding skip-one-unit transition edges, so that a deleted intermediate sound does not break the match.

Two warnings constrain the design. Unconstrained flooding spreads early mistakes to neighbouring matches, as the Lily OAEI systems found (Lily results for OAEI 2016). Real graphs also contain nodes with no counterpart, as studied in the dangling-entity work. Both motivate seeding the propagation with acoustic evidence and adding an explicit "no match" option.

The speech side supplies the gap. VoiceMixer is the closest existing use of similarity. It groups temporally adjacent frames by cosine similarity into variable-length segments, which is a one-step local operation within a single utterance. It builds no graph across speakers and computes no fixed point.

SKQVC separates identity as the time-average of the residual left after k-means quantization of WavLM features. That is a linear operation, and it depends on a large pretrained encoder.

S2CD obtains timbre with a speaker encoder that ignores frame order. We do the reverse and exploit frame order as the speaker-invariant signal.

The speaker-conditioned VQ-VAE of Unsupervised Acoustic Unit Discovery for Speech Synthesis removes identity by giving the decoder the speaker ID, so the bottleneck need not encode it. Nothing in that model forces a unit to mean the same thing across speakers. O_O-VC avoids separation altogether by synthesizing parallel data with a text-to-speech model. In doing so it inherits that model's biases and its need for text.

Research Program

Per-speaker clustering. For each speaker $s$, we extract frame features. The low-resource variant uses MFCCs. A second variant uses a low WavLM layer, to test whether the method still helps when strong features are available. Each speaker's frames are clustered into $K$ units, with $K$ somewhat larger than the expected phone inventory (50 to 100), so that allophones may split rather than merge.

Per-speaker graphs. We collapse each utterance into a unit sequence by merging repeated frames. Speaker $s$'s graph $G_s$ then has weighted directed edges for one-step and two-step transitions. Node attributes are quantities that change little across speakers: unit frequency and the shape of the duration distribution after normalizing by that speaker's speaking rate.

Alignment. We align each speaker graph to a consensus graph $G_c$. The consensus is initialized from one speaker and re-estimated as a weighted average after each round of alignment. Let $\sigma(i,a)$ be the similarity between speaker unit $i$ and consensus unit $a$. Let $w_s(i,j)$ and $w_c(a,b)$ be the normalized transition weights in the two graphs. Let $\sigma^0(i,a)$ be a weak acoustic prior, namely the cosine similarity of the two centroids after per-speaker mean subtraction. The update is

$$\sigma^{t+1} = \mathrm{Sinkhorn}\Big(\alpha\,\sigma^0 + \sigma^t + \beta \sum_{j,b}\big[w_s(i,j)\,w_c(a,b) + w_s(j,i)\,w_c(b,a)\big]\,\sigma^t(j,b)\Big),$$

where the components play these roles:

  • the $\sigma^t$ term is the self-propagation skip;
  • $\alpha$ and $\beta$ weight the acoustic prior against structural evidence;
  • Sinkhorn normalization is run against a consensus graph augmented with one dummy "unmatched" column, which absorbs dangling units and keeps the matching soft.

Content and identity codes. A frame's content code is the consensus unit its speaker-specific cluster maps to. The speaker's identity is the table of their centroids for each consensus unit. Following SKQVC, we keep the residual between each frame and its centroid as a time-varying prosody stream. Its role is to carry prosody and fine phonetic detail.

Graph construction cost. The graphs have roughly 100 nodes per speaker, so the explicit $N\times M$ similarity matrix that limits flooding on knowledge graphs is negligible here.

Decoder. For voice conversion, a vocoder-style decoder (HiFi-GAN) takes three inputs: the content units, the target speaker's centroid table, and a bottlenecked residual. To convert a voice, we swap in a new speaker's table. The table can be estimated for an unseen speaker from a few minutes of speech by running the alignment alone.

Baselines

The naive reference point is global k-means over all speakers' frames with the same features and the same total number of units. It shows how much speaker identity ordinary clustering leaks.

The strongest simple competitor is global k-means with per-speaker mean and variance normalization. This is the linear-subtraction idea at the heart of SKQVC's $R_{avg}$, applied at the feature level. It is the alternative most likely to match us on adult same-gender speech, which is why it is the main bar.

The key ablation is our own pipeline with $\beta=0$: per-speaker clusters matched to the consensus using only the acoustic prior and Sinkhorn. This isolates whether the transition graph contributes anything beyond matching centroid distributions. A second ablation removes the two-step edges, to test whether handling mismatched structure matters.

The strongest existing unit discovery method in the file is the speaker-conditioned VQ-VAE of Unsupervised Acoustic Unit Discovery. It reports 22.1% ABX error on ZeroSpeech 2019 English, against 22.7% for MFCCs and 29.5% for filterbanks. With pretrained features, the baseline is WavLM layer-6 k-means as used by SKQVC. For voice conversion on VCTK, the reference numbers are VoiceMixer's 1.47% speaker-classification accuracy on content embeddings, 20.92% WER and 3.75 zero-shot MOS. S2CD reports 4.22 MOS and 4.02 similarity MOS on unseen speakers. SKQVC reports 5.57% WER on LibriSpeech.

Fairness rules apply to every comparison:

  • Every clustering baseline gets the same features, the same total unit budget, and the same hyperparameter search budget (twenty configurations, selected on a held-out speaker set).
  • For voice conversion, we retrain all compared systems with the same HiFi-GAN decoder and the same training data. We do not rely on published numbers whose setups differ.
  • Published numbers serve only as sanity anchors.

Beating the baselines has a concrete meaning. With MFCC features, we must beat normalized global k-means and the VQ-VAE's 22.1% on across-speaker ABX. On the warp test we must keep a clear advantage that widens with warp strength. In conversion, we must reach lower speaker-classification accuracy on content codes at equal or better WER.

Why not just…?

Why not just normalize each speaker's features and cluster globally? This is the real threat. If speaker differences in MFCC space were mostly a shift and rescaling, normalization would remove them and the graph would be redundant. Vocal-tract-length differences, however, warp the frequency axis non-linearly. That predicts normalization should degrade as speakers grow more dissimilar, while the transition graph does not change. The controlled warp experiment below measures exactly this curve. If the two curves coincide, we will report that structure is unnecessary.

Why not just use WavLM or HuBERT k-means units, which are already fairly speaker-invariant? For high-resource English they may be, and our WavLM variant tests whether alignment still adds anything there. SKQVC itself argues that such k-means units discard prosody and fine phonetic detail and still carry speaker traits in their residual. The file also notes that these encoders shift domain in zero-resource languages. The ZeroSpeech surprise-language condition tests that case. The claim we will defend is narrower than "beats WavLM everywhere": the method gains most where pretrained features are absent or mismatched.

Why not just match per-speaker centroid distributions with optimal transport, with no graph? That is precisely the $\beta=0$ ablation. Distribution matching aligns sounds by where their centroids sit and how often they occur. Several vowels have similar frequencies and lie near one another, so this kind of matching should swap them under strong warps. Transition context is what distinguishes them. If the ablation matches the full method, the "structure as signature" thesis fails for speech.

Why not just train a speaker-conditioned VQ-VAE? Conditioning the decoder on the speaker removes the incentive for the bottleneck to encode identity. It gives no guarantee that unit 17 means the same sound for every speaker, and its reported ABX gain over raw MFCCs is small (22.1% versus 22.7%). We compare against it directly. We also test whether initializing its codebook from our consensus units helps, which would make the two methods complementary rather than competing.

Proposed Experiments

The decisive check is a controlled warp test with known answers. We take LibriSpeech and VCTK utterances and create pseudo-speakers by applying vocal-tract-length and formant warps of increasing strength. Each warped copy shares content with its original frame for frame. We then measure the normalized mutual information between unit sequences of original and warped copies, along with the purity of the unit-to-phone mapping, as a function of warp strength, for every method. Our method should be flat across warp strengths while normalized k-means declines. As in GMO's random-label test, this isolates whether structure alone carries correspondence when surface features are corrupted.

The second experiment is unit discovery on ZeroSpeech 2019 English and Indonesian, using the across-speaker ABX error. The anchor is the 22.1% VQ-VAE figure. We also evaluate on the ZeroSpeech 2020 surprise language, where IN-WAE from the WaveNet autoencoder work reports an ABX of 12.5. We present that number as context and do not expect an MFCC-only system to beat it.

The third experiment is voice conversion on VCTK with 10 held-out speakers. It reports four measures:

  • speaker-classification accuracy from content codes, where lower means less leakage;
  • WER with a fixed recognizer;
  • MOS and similarity MOS;
  • the drop in WER when the residual stream is removed, mirroring SKQVC's ablation, which saw WER rise from 5.57% to 12.93% without its residual.

VCTK speakers read shared prompts, so we also measure, on identical texts, how often two speakers' content sequences agree after time alignment. This gives a direct estimate of content invariance that needs no recognizer.

Finally, we stress the method's assumptions in two ways. We vary the amount of speech per speaker, from five minutes upward, to find where transition graphs become too sparse to align. We also record how often the dummy column absorbs units, a rate that should rise for accented speakers whose sound inventories differ. Either result would mark where the isomorphism assumption from PipEA stops holding for speech, and would tell us how far structural propagation can carry identity-content separation.