Distinguishable Speaker Anonymization based on Formant and Fundamental Frequency Scaling

Jixun Yao, Qing Wang, Yi Lei, Pengcheng Guo, Lei Xie, Namin Wang, Jie Liu

Abstract

Speech data on the Internet are proliferating exponentially because of the emergence of social media, and the sharing of such personal data raises obvious security and privacy concerns. One solution to mitigate these concerns involves concealing speaker identities before sharing speech data, also referred to as speaker anonymization. In our previous work, we have developed an automatic speaker verification (ASV)-model-free anonymization framework to protect speaker privacy while preserving speech intelligibility. Although the framework ranked first place in VoicePrivacy 2022 challenge, the anonymization was imperfect, since the speaker distinguishability of the anonymized speech was deteriorated. To address this issue, in this paper, we directly model the formant distribution and fundamental frequency (F0) to represent speaker identity and anonymize the source speech by the uniformly scaling formant and F0. By directly scaling the formant and F0, the speaker distinguishability degradation of the anonymized speech caused by the introduction of other speakers is prevented. The experimental results demonstrate that our proposed framework can improve the speaker distinguishability and significantly outperforms our previous framework in voice distinctiveness. Furthermore, our proposed method also can trade off the privacy-utility by using different scaling factors.

Introduction

The rapid growth of speech data sharing on social media platforms has heightened privacy risks, as voice recordings contain unique biometric identifiers. Speaker anonymization aims to remove these identifiers while maintaining the linguistic content and naturalness of the speech. A common approach in modern systems involves using "x-vector" embeddings, where a speaker's unique identity vector is replaced by an average or randomized embedding from a pool of other speakers.

However, this "averaging" process often leads to "identity collapse," where different anonymized speakers sound nearly identical. This loss of speaker distinguishability is a significant drawback for multi-speaker scenarios, such as anonymized conference calls or interviews, where the "who-is-who" relationship must be preserved. Building on their award-winning work from the VoicePrivacy 2022 challenge, the authors propose a new framework that avoids external speaker pools by directly manipulating the source speaker's own acoustic features.

Proposed Framework

The core innovation of this framework is the use of signal-processing-inspired scaling within a deep learning architecture to decouple identity concealment from identity merging.

Feature Extraction and Encoding

The system decomposes input speech into three primary streams:

  1. Linguistic Content: Bottleneck (BN) features are extracted using a WeNet-based ASR encoder. These features are designed to be speaker-invariant, capturing only the spoken words.
  2. Vocal Tract Characteristics: Formants (F1-F5) are extracted to represent the physical shape of the speaker's vocal tract.
  3. Prosody: The fundamental frequency (F0) is extracted to represent pitch and intonation.

Scaling Strategy

Instead of replacing these features with those of another speaker, the system applies a scaling factor ($\alpha$) to both the formants and the F0. This transformation shifts the perceived identity of the speaker without introducing foreign characteristics. The authors explored both gender-independent scaling (same factor for all) and gender-dependent scaling (e.g., scaling up for males and down for females) to maintain naturalness.

Acoustic Modeling and Synthesis

The scaled features are fed into an acoustic model consisting of CBHG-based encoders and an autoregressive decoder. This model predicts a Mel-spectrogram, which is then converted back into high-quality audio by a modified Multi-band HiFi-GAN vocoder.

Overview of the proposed speaker anonymization framework showing the feature extractor, scaling module, and acoustic model.

Experimental Evaluation

The framework was trained on LibriSpeech and LibriTTS datasets and evaluated using the standard VoicePrivacy Challenge benchmarks (LibriSpeech-test and VCTK-test).

Privacy and Utility Trade-offs

The scaling factor $\alpha$ serves as a tunable parameter for the privacy-utility trade-off. As shown in the results, increasing the deviation of $\alpha$ from 1.0 enhances privacy (measured by Equal Error Rate, EER) but can negatively impact intelligibility (measured by Word Error Rate, WER).

Voice Distinctiveness

A key metric in this study is the gain of voice distinctiveness ($G_{vd}$). Unlike x-vector methods that often result in highly negative $G_{vd}$ values (indicating identity collapse), the proposed scaling method keeps $G_{vd}$ closer to zero, successfully preserving the distinctiveness between different speakers even after anonymization.

Impact of scaling factors on privacy (EER), utility (WER), pitch correlation ($\rho^{F0}$), and voice distinctiveness (Gvd).

Results and Discussion

The experimental results highlight several key advantages of the proposed method:

  • Superior Privacy: The EER reached approximately 35% on LibriSpeech-test, a significant improvement over the 20% achieved by previous x-vector-based systems.
  • Maintained Intelligibility: The WER remained low (around 4.3% for LibriSpeech), demonstrating that the anonymization process does not significantly degrade machine recognition of the speech content.
  • Preserved Prosody: A high pitch correlation ($\rho^{F0} \approx 0.8$) indicates that the original speaker's emotional intonation is well-preserved, which is crucial for natural-sounding communication.

Conclusion

By moving away from averaged speaker embeddings and toward direct formant and F0 scaling, this research provides a robust solution to the problem of identity collapse in speaker anonymization. The framework offers a controllable trade-off between privacy and utility while ensuring that anonymized voices remain distinct. Future work will focus on exploring non-linear transformations and developing a fully end-to-end neural pipeline to further enhance security against potential de-anonymization attacks.

Summary updated

Abstract

Speech data on the Internet are proliferating exponentially because of the emergence of social media, and the sharing of such personal data raises obvious security and privacy concerns. One solution to mitigate these concerns involves concealing speaker identities before sharing speech data, also referred to as speaker anonymization. In our previous work, we have developed an automatic speaker verification (ASV)-model-free anonymization framework to protect speaker privacy while preserving speech intelligibility. Although the framework ranked first place in VoicePrivacy 2022 challenge, the anonymization was imperfect, since the speaker distinguishability of the anonymized speech was deteriorated. To address this issue, in this paper, we directly model the formant distribution and fundamental frequency (F0) to represent speaker identity and anonymize the source speech by the uniformly scaling formant and F0. By directly scaling the formant and F0, the speaker distinguishability degradation of the anonymized speech caused by the introduction of other speakers is prevented. The experimental results demonstrate that our proposed framework can improve the speaker distinguishability and significantly outperforms our previous framework in voice distinctiveness. Furthermore, our proposed method also can trade off the privacy-utility by using different scaling factors.
BibTeX
@inproceedings{yao2023distinguishable,
  author = {Jixun Yao and Qing Wang and Yi Lei and Pengcheng Guo and Lei Xie and Namin Wang and Jie Liu},
  title = {{Distinguishable Speaker Anonymization based on Formant and Fundamental Frequency Scaling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=UHH5iLepwJZ},
}
Figure