NTU-NPU System for Voice Privacy 2024 Challenge

Nikita Kuzmin, Hieu-Thi Luong, Jixun Yao, Lei Xie, Kong Aik Lee, Eng Siong ChngarXiv 2024 · DOI · arXiv

Introduction

The rapid proliferation of voice-controlled AI has heightened concerns regarding "voice leaking," where a speaker's unique identity is extracted from audio data. Speech anonymization aims to mitigate this risk by stripping away identity-revealing features—such as timbre and specific pitch patterns—while preserving the underlying linguistic content. The 2024 Voice Privacy Challenge introduced a significant new requirement: Emotion Preservation. Anonymized speech must not only hide the speaker's identity but also retain the emotional nuance of the original message. The NTU-NPU team addressed this "privacy-utility-emotion" trilemma by enhancing existing baseline systems with novel techniques like Mean Reversion F0 and specialized emotion embeddings.

Methodology

The team's approach involved refining two primary baseline systems, B3 and B5, to better balance privacy and emotional fidelity.

Enhancing Baseline B3: Generative Anonymization

The B3 baseline was modified to incorporate emotional features and more robust speaker representations. The system extracts content, prosody, and identity features separately to allow for independent manipulation.

  • Emotion Embeddings: A fine-tuned Wav2Vec2 model, trained on the MSP-Podcast dataset, was integrated to extract emotional features. These embeddings were provided to the FastSpeech2 synthesis component to ensure the anonymized output mirrored the source's emotional state.
  • Advanced Speaker Embedders: Standard Global Style Tokens (GST) were replaced with WavLM and ECAPA2 models to capture identity more effectively before masking.
  • Anonymization Strategies: The system employed random speaker selection and cross-gender conversion (e.g., mapping male voices to female targets) to maximize the identity gap between the source and the output.
Architecture of the enhanced B3 system incorporating emotion embeddings and anonymization modules.

Enhancing Baseline B5: Signal Processing and Mean Reversion

For the B5 baseline, the team introduced Mean Reversion F0, a novel technique for pitch manipulation designed to reduce the speaker's unique prosodic signature.

  • Mean Reversion F0: Instead of simple pitch shifting, this method calculates a weighted average between the original fundamental frequency (F0) and a moving average. This "pulls" the pitch toward a neutral mean, smoothing out the unique dynamic range that often characterizes an individual's voice.
  • Stochastic Masking: To further obfuscate the speaker's "voiceprint," Additive White Gaussian Noise (AWGN) was added to the pitch contour.
Visualization of Mean Reversion F0 (a) and Mean Reversion F0 with added white Gaussian noise (b).

Experimental Setup

The study utilized several standard datasets for training and evaluation:

  • LibriTTS: Used for training the core synthesis and Automatic Speech Recognition (ASR) models.
  • Librispeech: Employed to evaluate utility via Word Error Rate (WER).
  • MSP-Podcast (v1.7): Essential for training the emotion embedding extractor.
  • VoxPopuli: Used for training linguistic representation components in the B5 system.

Results and Discussion

The evaluation focused on the trade-offs between privacy (Equal Error Rate, EER), utility (WER), and emotion preservation (Unweighted Average Recall, UAR).

Key Insights

  • Emotion-Privacy Trade-off: Integrating emotion embeddings significantly improved UAR (jumping from ~30% to over 45%) but often resulted in lower EER. This suggests that emotional expression is intrinsically linked to speaker identity, creating a "leakage" path.
  • Effectiveness of Mean Reversion: The Mean Reversion F0 technique was highly successful in enhancing privacy. By smoothing pitch "peaks and valleys," the system achieved EERs exceeding 40% while maintaining acceptable speech quality.
  • Cross-Gender Conversion: Forcing the system to select a target speaker of the opposite gender consistently yielded the best privacy scores and, in some cases, improved ASR performance.

Performance Metrics

The team developed configurations for specific privacy conditions:

  • Condition EER1 (10-20%): Achieved an EER of ~10.5% with a low WER of 4.6%, prioritizing utility.
  • Condition EER4 (>40%): Utilizing Mean Reversion F0 and 10-dB AWGN, the system reached an EER of 43.15%, effectively thwarting speaker recognition attempts with only a slight increase in WER (4.4%).

Conclusion

The NTU-NPU submission demonstrates that while preserving emotion in anonymized speech is challenging, it is achievable through targeted architectural enhancements. However, the authors noted high volatility in results based on hardware and random seeds, indicating a need for more stable anonymization metrics. Future research will focus on better disentangling emotion from identity to prevent identity leakage through emotional prosody.

Summary updated

Abstract

In this work, we describe our submissions for the Voice Privacy Challenge 2024. Rather than proposing a novel speech anonymization system, we enhance the provided baselines to meet all required conditions and improve evaluated metrics. Specifically, we implement emotion embedding and experiment with WavLM and ECAPA2 speaker embedders for the B3 baseline. Additionally, we compare different speaker and prosody anonymization techniques. Furthermore, we introduce Mean Reversion F0 for B5, which helps to enhance privacy without a loss in utility. Finally, we explore disentanglement models, namely $β$-VAE and NaturalSpeech3 FACodec.
BibTeX
@misc{kuzmin2024ntu,
  author = {Nikita Kuzmin and Hieu-Thi Luong and Jixun Yao and Lei Xie and Kong Aik Lee and Eng Siong Chng},
  title = {{NTU-NPU System for Voice Privacy 2024 Challenge}},
  year = {2024},
  eprint = {2410.02371},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.02371},
}
Figure