TL;DR
Even if you completely alter your pitch and vocal timbre, the unique rhythm of how you pronounce individual sounds can still give you away. This paper demonstrates that phoneme durations act as a significant privacy leak in state-of-the-art voice anonymization systems, allowing speaker re-identification with surprising accuracy if enough speech data is available.
What Problem Does This Paper Address?
Voice anonymization aims to strip speaker-identifying traits (like identity, gender, and accent) from an audio recording while keeping the linguistic content intact. Modern approaches typically achieve this by altering the speaker's pitch, energy, or neural speaker embeddings (like x-vectors).
However, the authors point out a massive blind spot: speech temporal dynamics. Most current anonymization systems leave the speaker's speech rate and individual phoneme durations completely untouched. This paper investigates whether an attacker can exploit these subtle timing patterns to bypass voice anonymization and successfully re-identify a speaker.
Methods
The authors developed an automatic speaker verification (ASV) framework that acts as an "attacker," attempting to match speakers based only on the timing of their speech.
They proposed two distinct distance metrics to compare the temporal footprints of two speakers:
- Metric 1 ($\rho_1$): The cosine distance between two vectors representing the average duration of each phoneme class.
- Metric 2 ($\rho_2$): A min-ratio based distance metric that compares the relative duration differences across phoneme classes.
They experimented with two phoneme configurations: a standard 39-phoneme set (based on ARPAbet) and an expanded 336-phoneme set that accounts for word position and lexical stress.
To evaluate privacy protection, they tested these metrics against original speech and speech processed by two state-of-the-art Speaker Anonymization Systems (SAS):
- SAS-1: A system that alters pitch, energy, and speaker identity embeddings but explicitly preserves original phoneme durations.
- SAS-2: A cascaded Automatic Speech Recognition and Text-to-Speech (ASR-TTS) system that synthesizes an entirely new voice, naturally altering phoneme durations in the process.
Datasets
The experiments were conducted using the LibriSpeech corpus, a standard benchmark of read English audiobooks:
- LibriSpeech-train-960: Used for the initial baseline analysis on original speech across 2,338 speakers.
- LibriSpeech-train-clean-360: Processed through SAS-1 and SAS-2 to evaluate anonymization resilience.
- LibriSpeech test dataset: Used to establish standard baseline verification performance.
Key Insights and Ideas
- Phonemes carry more identity than global speed: The unique rhythm of how a speaker stretches or shortens specific sounds (e.g., holding vowels longer vs. shortening consonants) contains far more speaker-identifying information than their overall global speech rate.
- Data aggregation fuels the attack: While a single short utterance might not reveal a speaker's temporal footprint, aggregating data across multiple utterances (e.g., 20 to 60 sentences) allows the duration-based attacker to build a highly accurate profile.
- TTS systems aren't perfectly safe: Even when using a full ASR-TTS pipeline (SAS-2) that generates synthetic phoneme durations, a lingering identity leak remains over long audio sequences. The authors hypothesize that the underlying text content or reading style of the source book still biases the temporal statistics.
Key Results
The duration-based ASV system proved remarkably effective at identifying speakers, especially when given more data to analyze.
- On Original Data: Using the proposed $\rho_2$ metric with 38 phonemes, the Equal Error Rate (EER) dropped to just 2.5% when averaging over 60 utterances, showing that phoneme duration alone is a highly discriminative biometric.
- Defeating SAS-1 (Preserved Durations): Because SAS-1 leaves timing intact, the attacker easily bypassed its voice alterations, achieving a low EER of 7.0% with 60 utterances.
- Challenging SAS-2 (Altered Durations): As expected, the ASR-TTS system provided much stronger privacy protection. However, when the attacker aggregated 60 utterances, the EER was still surprisingly low at 26.3%, confirming that a partial privacy leak persists.
Limitations and Future Work
The primary limitation is that this study was conducted entirely on read speech (audiobooks), where timing is naturally more structured and influenced by the text. The authors note that future work must verify if these temporal dynamics hold up as strongly in spontaneous speech, where pauses, filler words, and conversational context introduce more variability.
Additionally, they plan to use these insights to engineer better anonymization systems by integrating explicit temporal dynamics normalization to completely erase a speaker's rhythmic signature.
Summary updated