Investigation of perception inconsistency in speaker embedding for asynchronous voice anonymization

Rui Wang, Liping Chen, Kong Aik Lee, Zhengpeng Zha, Zhenhua LingarXiv 2025 · DOI · arXiv

Abstract

Given the speech generation framework that represents the speaker attribute with an embedding vector, asynchronous voice anonymization can be achieved by modifying the speaker embedding derived from the original speech. However, the inconsistency between machine and human perceptions of the speaker attribute within the speaker embedding remains unexplored, limiting its performance in asynchronous voice anonymization. To this end, this study investigates this inconsistency via modifications to speaker embedding in the speech generation process. Experiments conducted on the FACodec and Diff-HierVC speech generation models discover a subspace whose removal alters machine perception while preserving its human perception of the speaker attribute in the generated speech. With these findings, an asynchronous voice anonymization is developed, achieving 100% human perception preservation rate while obscuring the machine perception. Audio samples can be found in https://voiceprivacy.github.io/speaker-embedding-eigen-decomposition/.

Introduction

In the era of ubiquitous voice-activated technology and biometric security, voice privacy has become a critical concern. Traditional voice anonymization methods are typically "synchronous," meaning they alter the voice in a way that is perceptible to both humans and machines. However, there is a growing need for asynchronous voice anonymization: a technique that renders speech unrecognizable to automatic speaker verification (ASV) systems while maintaining perfect recognizability for human listeners. This study investigates the perceptual gap between humans and machines by analyzing the speaker embedding—a high-dimensional vector representing a person's unique vocal identity.

Methodology

The researchers proposed a framework that disentangles speech into prosody, linguistic content, and speaker attributes. The core of the method lies in the modification of the speaker embedding vector ($x$) before it enters the waveform generation stage.

Framework for asynchronous voice anonymization via speaker embedding modification.

To identify which parts of the embedding are critical for different types of perception, the team performed an eigen-decomposition on speaker embeddings from the LibriSpeech dataset. This allowed them to categorize the embedding space into three distinct subspaces:

  1. Primary Subspace ($S_p$): Dimensions with the highest eigenvalues, containing core identity features.
  2. Secondary Subspace ($S_s$): Dimensions where the mathematical importance (eigenvalue delta) fluctuates and then increases again.
  3. Residual Subspace ($S_r$): The remaining low-importance dimensions.

By systematically zeroing out coefficients in these subspaces, the researchers could observe the impact on machine models (ECAPA-TDNN, ResNet) and human listeners.

Key Insights and Results

The study revealed a significant inconsistency in how humans and machines process speaker information. The most striking discovery was the role of the Secondary Subspace.

The Machine-Human Perception Gap

As shown in the experimental results, modifying the primary subspace causes an immediate and sharp decline in both machine and human recognition. This suggests that both humans and machines rely on these dominant features for identification. However, the secondary subspace behaves differently. Removing dimensions in this subspace significantly increases the Equal Error Rate (EER) for ASV systems (indicating successful anonymization) while the human perception rate remains at a perfect 100%.

Performance metrics (EER, Human Perception Rate, and WER) across different subspaces for FACodec and Diff-HierVC models.

Quantitative Performance

In the FACodec model, the proposed method achieved an EER of 8.76% for machine models while maintaining a 100% human perception rate. This outperforms state-of-the-art adversarial perturbation methods, which often suffer from a trade-off where machine obscurity comes at the cost of human recognizability (e.g., dropping to ~60% as seen in the adversarial baseline).

Residuals and Linguistic Content

The study also found that the residual subspace, while least important for identity, carries essential linguistic information. Modifying this subspace did not significantly aid anonymization but did lead to a spike in the Word Error Rate (WER), making the speech harder for speech-to-text models like Whisper to transcribe accurately.

Conclusion

This research demonstrates that asynchronous voice anonymization is achievable by exploiting the perceptual inconsistencies between humans and machines within the speaker embedding space. By targeting the "secondary subspace," it is possible to create voices that are invisible to AI surveillance but remain perfectly familiar to human listeners. Future work will focus on generalizing the identification of these subspaces across different speech models and ensuring robustness against potential re-identification attacks.

Summary updated

Abstract

Given the speech generation framework that represents the speaker attribute with an embedding vector, asynchronous voice anonymization can be achieved by modifying the speaker embedding derived from the original speech. However, the inconsistency between machine and human perceptions of the speaker attribute within the speaker embedding remains unexplored, limiting its performance in asynchronous voice anonymization. To this end, this study investigates this inconsistency via modifications to speaker embedding in the speech generation process. Experiments conducted on the FACodec and Diff-HierVC speech generation models discover a subspace whose removal alters machine perception while preserving its human perception of the speaker attribute in the generated speech. With these findings, an asynchronous voice anonymization is developed, achieving 100% human perception preservation rate while obscuring the machine perception. Audio samples can be found in https://voiceprivacy.github.io/speaker-embedding-eigen-decomposition/.
BibTeX
@misc{wang2025investigation,
  author = {Rui Wang and Liping Chen and Kong Aik Lee and Zhengpeng Zha and Zhenhua Ling},
  title = {{Investigation of perception inconsistency in speaker embedding for asynchronous voice anonymization}},
  year = {2025},
  eprint = {2510.05718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.05718},
  howpublished = {arXiv},
}

Loading related content...

Figure