A person's speech carries an identity signal that survives almost every attempt to disguise it: not the sound of the voice, but its timing. How long you hold each vowel, how quickly you release a consonant, how you pace syllables against each other — these habits come from learned motor control and persist even when pitch and vocal-tract acoustics are scrambled by voice-anonymization systems. Recent work has turned this into a working biometric: a model that reads only phoneme durations can verify speakers at a 1.8% equal error rate (EER) on original speech and 3.9% on anonymized speech Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems. But every such system sits on top of a fragile pipeline: an upstream aligner must first decide, as hard discrete choices, where each phoneme begins and ends. When that decision is wrong — and it will be wrong on anonymized, noisy, or spontaneous audio — the error propagates unconditionally into the duration features, with no mechanism downstream to reconsider it. This is exactly the cascading-error failure that the pipeline-error-propagation literature has diagnosed and partially fixed in text processing, and it is the explicitly acknowledged blind spot of the strongest temporal-fingerprinting systems.
The proposal here is to import the soft-evidence remedy from that literature into temporal fingerprinting, and in doing so to deliver the first systematic characterization of how upstream alignment errors degrade temporal speaker verification. Concretely: replace hard phoneme durations with soft, posterior-weighted expected durations, train the verification back-end on these, and measure the full degradation curve of EER against controlled alignment corruption. The applied stakes are real — this is the attack channel that current anonymization defenses do not close The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymization — but the deeper contribution is a case study in error propagation across a discrete boundary in a modality (continuous audio) where the boundary decisions are millisecond-graded rather than token-graded.
Core Claim
The claim is that a speaker-verification system built on soft durations degrades gracefully as upstream alignment quality falls, while the existing hard-duration architecture degrades sharply, and that the soft representation recovers most of the gap at no cost on clean alignments. The mechanism is simple. Let $P_t(p)$ be the posterior probability, from a speaker-independent acoustic model, that audio frame $t$ belongs to phoneme $p$. The existing systems compute a hard duration $d_p$ by counting frames between Viterbi boundaries and encoding each segment as a one-hot vector scaled by that count Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems. We instead compute
$$\tilde{d}_p = \sum_{t=1}^{T} w_t \, P_t(p),$$
where $w_t$ is an optional per-frame confidence weight (for example, inversely related to the entropy of $P_t$). Hard durations are the degenerate case where $P_t$ is one-hot between committed boundaries. The key property is that under zero-mean boundary jitter, the expected value of $\tilde{d}_p$ is unchanged: a boundary that wobbles by 20 ms shifts probability mass between two adjacent phonemes rather than reassigning 20 ms wholesale, so the duration distribution a speaker profile is built from stays close to its true value, while a hard count absorbs the full error. This is the audio analogue of what Relieving the Computational Bottleneck: Joint Inference for Event Extraction with High-Dimensional Features demonstrated for text — passing continuous SVM margins into a joint Markov Logic Network instead of hard binary decisions bought a 10-point F1 improvement on complex regulation events in BioNLP'13 — and what Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction did by treating evidence relevance as a latent variable rather than a retriever's hard ranking. Nobody has applied this principle to phonetic segmentation, and the two strongest temporal-fingerprinting papers both flag it as open: the ICASSP system forces alignments with ground-truth transcripts and never models alignment failure Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization, and the Interspeech follow-up states explicitly that the impact of ASR errors on phonetic alignment is crucial and untested for in-the-wild attacks Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems. Our proposal is precisely that missing study plus the fix.
The closest prior work at the intersection uses continuous phoneme posteriorgrams, but for a different purpose: Rhythm-Flexible Voice Conversion without Parallel Data Using Cycle-GAN over Phoneme Posteriorgram Sequences transforms posteriorgram sequences between speakers to convert speaking tempo, and it does not extract any biometric fingerprint or model duration statistics explicitly. We reuse the posteriorgram idea as an input representation but aggregate it into interpretable per-phoneme duration profiles for verification — a different output, objective, and failure analysis. On the pipeline side, SegReConcat: A Data Augmentation Method for Voice Anonymization Attack is the only paper that studies upstream-segmentation errors in a speech attack pipeline, but it treats the problem as one of training data (rearranging word slices so the verifier cannot overfit to syntax) and works strictly at the word level with hard Whisper-medium timestamps; it neither quantifies error propagation nor changes the representation handed downstream. Our approach changes the interface itself.
Research Program
Three components. First, the soft-duration extractor: a speaker-independent acoustic model (a phoneme recognizer of the kind used to produce posteriorgrams, or a self-supervised encoder fine-tuned for frame-level phoneme classification, for which Problem Agnostic Speech Encoder is a candidate backbone) produces $P_t$, from which we build both the 39-phoneme and 336 context-dependent profiles of Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization and a sequence input for an ECAPA-TDNN styled on the A2 model of Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems. Second, an alignment-corruption simulator with three error families: transcript-level ASR errors (substitutions, insertions, deletions at controlled rates, followed by forced alignment on the corrupted transcript), boundary jitter (Gaussian perturbation of boundary times at standard deviations of 10 to 80 ms), and real ASR output — Whisper-medium transcriptions of the kind used in SegReConcat — applied to original and anonymized audio, so the corruption is organic rather than synthetic. Third, the propagation analysis: EER as a function of corruption severity for hard versus soft representations, fit as a degradation curve, plus attribution of which error types (boundary drift versus phoneme confusion versus deletion) drive the damage. As a secondary analysis we will examine whether vector-quantized anonymization systems (B5, T12-5 in VoicePrivacy 2024), which SegReConcat found shatter temporal continuity, corrupt soft durations differently than hard ones — the anonymization-vs-fingerprint side of the same error-propagation coin.
We deliberately keep a collapsed single-pass architecture (one network doing frame-level phoneme tagging and speaker embedding jointly, in the spirit of Emotion-Cause Pair Extraction as Sequence Labeling Based on A Novel Tagging Scheme and Biomedical Event Extraction as Sequence Labeling, which dissolved two-stage NLP pipelines into one-pass taggers) as an exploratory arm rather than the headline. Those methods exploit discrete clause and token boundaries that speech does not have, and collapsing everything would discard the interpretable per-phoneme duration profile that matters for privacy auditing and forensic use, where one needs to say which timing habit leaked.
Proposed Experiments
Training and evaluation on LibriSpeech: train-960 (2,338 speakers) for the speaker-classification front-end, test-clean for verification with the 8-enrollment-utterance protocol of Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems, so our clean-condition numbers are directly comparable to its reported 1.8% EER on original speech and 3.9% EER on speech anonymized by the VoicePrivacy 2024 GAN baseline B3. The main experiment sweeps the corruption simulator and reports EER-versus-severity curves for each representation. The application experiment runs the attack on VoicePrivacy 2024 anonymization systems (B3, plus VQ-based B5 and T12-5) with alignments derived from real ASR output on the anonymized audio rather than ground-truth transcripts — the first evaluation of temporal fingerprinting under the conditions its own authors call crucial. A spontaneous-speech probe uses the TEVOID corpus of Zurich German Speaker idiosyncratic rhythmic features in the speech signal, whose 8 speakers are too few for verification claims but sufficient to test whether the vocalic-interval rhythm metrics found there to carry between-speaker variance (%V, nPVI-V, VarcoPeak) remain stable under alignment corruption in their soft form — the regime where, as that paper documents, pacing varies within a speaker and standardized phonetic sequences disappear.
Baselines
The naive reference point is a speaking-rate-only profile: a single tempo scalar per speaker (syllables per second). The literature already tells us this should sit near chance, because identity lives in relative durations rather than global pace — the $\rho_2$ ratio metric of Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization exists precisely because mean durations need to be compared to global norms, and the rhythm-embedding work of Phoneme Duration Modeling Using Speech Rhythm-Based Speaker Embeddings for Multi-Speaker Speech Synthesis shows tempo alone is not the signal. We include it to calibrate how much of the soft-duration performance is just pacing. The strongest existing competitor is the hard-duration pipeline of Tomashenko et al., specifically the A2 ECAPA-TDNN over context-dependent duration sequences, which reports an EER of 1.8% on original LibriSpeech test-clean with 8 enrollment utterances and 3.9% on B3-anonymized speech Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems; its metric-based sibling reports 2.5% EER with 60 enrollment utterances on original speech using the $\rho_2$ distance, 7.1% against the timing-preserving anonymizer SAS-1, and 26.3% against the timing-regenerating ASR-TTS cascade SAS-2 Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization. We will run it under our corruption simulator at every severity level, on identical audio, enrollment counts, and scoring code, with the same tuning budget (matched parameter count and training data). Beating it means a strictly lower EER at every nonzero corruption severity while losing no more than half a point of EER at zero corruption on original speech. The most important baseline is the ablation that isolates our ingredient: the same A2 architecture, same corrupted training regime, but with hard durations — that is, a model made robust to alignment noise purely by seeing noisy alignments during training. If soft durations beat this noise-trained hard model only on the error families it was trained on but not on held-out families, the contribution of softness per se is generalization across error types, and we will say so plainly. Finally, on the anonymization attack we include the augmentation competitor, SegReConcat-style word rearrangement plus SpecAugment, since SegReConcat showed SpecAugment alone is insufficient and its rearrangement-plus-concatenation strategy cut EER against system T8-5 from 37.39% to 26.51%; the fair comparison runs both on the same VoicePrivacy 2024 systems with the same ASR-derived alignments, since SegReConcat's gains might subsume or complement ours.
Why not just…?
Why not just use a better ASR system or aligner? Because the audio we must attack is exactly the audio where aligners fail. Anonymization pipelines themselves degrade ASR — the VoicePrivacy baseline introduced a 19% to 67% relative increase in word error rate Introducing the VoicePrivacy Initiative — and anonymized speech is out-of-domain for any aligner trained on natural speech, so residual errors are unavoidable, not a matter of buying a bigger model. Our degradation curves make this quantitative: whatever the aligner's floor error rate is, the question is whether the fingerprint extractor tolerates it, and that question is independent of how good the aligner gets. That said, we will run the experiment with two aligners of different quality so the claim is not tied to one weak front-end.
Why not just train the existing hard-duration model on corrupted alignments? This is the real threat, and we treat it as the key ablation rather than arguing around it. Simulated-noise training is cheap and might close most of the gap on the specific corruption family used in training. Our hypothesis is that it will not generalize: a hard count absorbs the full magnitude of every boundary error, so the model can only learn to tolerate the noise distribution it saw, whereas soft durations are approximately unbiased under zero-mean jitter by construction. The decisive check trains each model on one error family and tests on a held-out family (jitter-trained, tested on real ASR errors, and vice versa). If the noise-trained hard model matches soft durations across families, we will report that the soft representation is unnecessary and the contribution reduces to the corruption-robust training recipe plus the first propagation characterization — still worth having, but we will not oversell the representation.
Why not just use spectral speaker embeddings (x-vectors)? Because they fail in the regime that motivates this work. Under anonymization that alters pitch and vocal-tract characteristics, spectral verification degrades while duration-based attack holds at 3.9% EER against B3 and 7.1% against SAS-1 Exploiting Context-dependent Duration Features for Voice Anonymization Attack SystemsAnalysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization; the VoicePrivacy 2020 baseline showed EERs of 26% to 37% once enrollment is anonymized Introducing the VoicePrivacy Initiative; and Phoneme Duration Modeling Using Speech Rhythm-Based Speaker Embeddings for Multi-Speaker Speech Synthesis shows x-vectors do not encode rhythmic information, with worse duration-prediction error than rhythm embeddings. Forensic evidence also points the other way: duration ratios stay identifiable when noise obscures spectral detail A forensic phonetic investigation into the duration and speech rate. Temporal and spectral channels are complements, and our study is about the temporal channel's pipeline fragility.
Why not just collapse everything into one end-to-end network? The collapse strategy that worked for emotion-cause extraction Emotion-Cause Pair Extraction as Sequence Labeling Based on A Novel Tagging Scheme and biomedical event extraction Biomedical Event Extraction as Sequence Labeling removes the discrete interface entirely, which in principle eliminates the propagation problem rather than softening it. We keep it as an exploratory arm because it could win on raw EER. Two things hold us back from making it the headline: those methods rely on discrete symbolic boundaries that continuous audio lacks, so the "collapse" in speech is really an uninterpretable latent — and for privacy auditing and forensic analysis, the value of the duration profile is that one can point at which phoneme's timing carries identity. A black-box that verifies well but cannot attribute the leak answers a weaker question. If the collapsed model dominates empirically, we will report it and frame the soft-duration model as the interpretable middle ground.
Decisive Checks
The proposal succeeds if three results hold: soft durations keep clean-condition EER within half a point of the 1.8% reported for the hard A2 model Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems; the EER degradation curve of the soft model is flatter than the hard model's across at least two error families, including real ASR output on anonymized audio; and the held-out-family test separates softness from noise-training. It fails informatively if noise-trained hard durations generalize as well — in which case the deliverable is the first measured characterization of alignment-error propagation in temporal speaker verification, a result the field currently has zero data on, and a training recipe that makes in-the-wild temporal attacks practical. Either outcome moves past a literature in which the strongest systems assume gold alignments and the error-propagation problem has been named but never opened.
References
- Exploiting Context-dependent Duration Features for Voice Anonymization Attack Systems
- The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymization
- Relieving the Computational Bottleneck: Joint Inference for Event Extraction with High-Dimensional Features
- Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction
- Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization
- Rhythm-Flexible Voice Conversion Without Parallel Data Using Cycle-GAN Over Phoneme Posteriorgram Sequences
- SegReConcat: A Data Augmentation Method for Voice Anonymization Attack
- Emotion-Cause Pair Extraction as Sequence Labeling Based on A Novel Tagging Scheme
- Biomedical Event Extraction as Sequence Labeling
- Speaker idiosyncratic rhythmic features in the speech signal
- Phoneme Duration Modeling Using Speech Rhythm-Based Speaker Embeddings for Multi-Speaker Speech Synthesis
- A forensic phonetic investigation into the duration and speech rate