Distinctive and Natural Speaker Anonymization via Singular Value Transformation-Assisted Matrix

Jixun Yao, Qing Wang, Pengcheng Guo, Ziqian Ning, Lei XieIEEE/ACM Trans. ASLP 2024 · OpenReview

TL;DR

This paper introduces a novel speaker anonymization technique that uses Singular Value Decomposition (SVD) to transform speaker identity representations, effectively concealing a speaker's identity while preserving speech naturalness and distinctiveness. This method aims to overcome limitations of existing approaches that often degrade speech quality or fail against sophisticated attackers.

What Problem Does This Paper Address?

In today's digital world, speech data is abundant, making it vulnerable to theft and misuse. Speaker anonymization is crucial for protecting personal privacy by removing identifying information from speech while keeping the content and naturalness intact. Current methods often rely on utterance-level speaker embeddings, which can lead to a loss of distinctiveness, unnatural speech, and privacy leakage, especially when attackers have advanced knowledge of the anonymization system. The VoicePrivacy Challenge (VPC) has highlighted these issues, driving the need for more robust and effective anonymization techniques.

Methods

The proposed approach introduces a "Singular Value Transformation-Assisted Matrix" for speaker anonymization. Instead of using a single utterance-level vector, the system extracts frame-level speaker vectors from a pre-trained ASV model. These frame-level vectors are then processed through an attention mechanism to create a "speaker-score matrix" and "speaker-related tokens." The speaker-score matrix acts as a weight for these tokens, representing the speaker's identity.

The core innovation lies in the use of Singular Value Decomposition (SVD) on this speaker-score matrix. SVD decomposes the matrix into orthonormal eigenvectors and singular values. The singular values are then non-linearly transformed (specifically, a logarithmic transformation is applied) to modify the speaker identity representation. This transformed matrix is then recomposed with the eigenvectors to create the "singular value transformation-assisted matrix." This process effectively "anonymizes" the speaker identity representation without introducing information from other speakers, thus preserving distinctiveness.

The system also incorporates a soft content extractor (based on HuBERT) for linguistic content and a prosody extractor, aiming to disentangle content, prosody, and speaker identity through Mutual Information (MI) minimization. A reconstruction loss ensures the original speech can be synthesized, and an auxiliary classification loss helps the speaker-score matrix focus on relevant speaker information.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Training: LibriSpeech-train-clean-100, LibriSpeech-train-other-500, LibriTTS-train-clean-100, LibriTTS-train-other-500.
  • Development: LibriSpeech-dev-clean, VCTK-dev.
  • Testing: LibriSpeech-test-clean, VCTK-test.
  • Speaker Verification Pre-training: VoxCeleb2.

The evaluation is conducted on official VoicePrivacy Challenge (VPC) development and test datasets, which include speakers from LibriSpeech, VCTK, and other corpora.

Key Insights and Ideas

  • Frame-level vs. Utterance-level Embeddings: The paper argues that frame-level speaker vectors, when processed appropriately, are more effective for anonymization than traditional utterance-level vectors. This allows for capturing more nuanced temporal variations in speaker identity.
  • SVD for Identity Transformation: Using SVD to decompose and then recompose the speaker-score matrix is a novel way to modify speaker identity representations. This approach directly manipulates the underlying structure of the identity information, rather than just averaging or scaling it.
  • Preserving Distinctiveness: The SVD-based transformation is designed to avoid introducing external speaker information, a common pitfall in previous methods that leads to degraded distinctiveness.
  • Disentanglement for Naturalness: The integration of soft content extraction and MI constraints helps to disentangle linguistic content, prosody, and speaker identity, which is crucial for maintaining speech naturalness.

Key Results

The proposed system demonstrates strong performance across various metrics:

  • Privacy: In the "Semi-informed" attack scenario (the most challenging), the system achieves the highest Equal Error Rate (EER) among compared systems, indicating superior privacy protection. For example, on the test set, the weighted average EER is 39.74%, significantly better than baselines like B1.a (52.11%) and B1.b (51.27%).
  • Intonation: The pitch correlation (pF0) metric shows excellent preservation, with average scores of 0.84 on the test set, far exceeding the VPC minimum requirement of 0.3.
  • Distinctiveness: The Gain of Voice Distinctiveness (Gvd) metric is close to 0 (e.g., -0.98 on the test set), indicating that the original distinctiveness is well-preserved.
  • Intelligibility: The Word Error Rate (WER) is low, with an average of 3.35% for LibriSpeech and 9.59% for VCTK, demonstrating good linguistic content preservation.
  • Subjective Evaluation: MOS scores indicate that the proposed system achieves higher naturalness than all baseline systems, and SMOS scores suggest superior privacy protection.

Limitations and Future Work

  • Language Dependency: A significant limitation is that the current approach is primarily designed for English speakers due to the use of a language-dependent content extractor trained on English data. The authors suggest that an ideal anonymization system should be language-independent.
  • Complexity: While the paper compares complexity, the practical implications of the SVD transformation and attention mechanisms in real-time applications might warrant further investigation.

Future work could focus on extending the system to handle multiple languages and exploring more efficient methods for disentangling speech attributes.

Summary updated

Abstract

Speaker anonymization is an effective privacy protection solution that aims to conceal speaker's identity while preserving the naturalness and distinctiveness of the original speech. Mainstream approaches use an utterance-level vector from a pre-trained automatic speaker verification (ASV) model to represent speaker identity, which is then averaged or modified for anonymization. However, these systems suffer from deterioration in the naturalness of anonymized speech, degradation in speaker distinctiveness , and severe privacy leakage against powerful attackers. To address these issues and especially generate more natural and distinctive anonymized speech, we propose a novel speaker anonymization approach that models a matrix related to speaker identity and transforms it into an anonymized singular value transformation-assisted matrix to conceal the original speaker identity. Our approach extracts frame-level speaker vectors from a pre-trained ASV model and employs an attention mechanism to create a speaker-score matrix and speaker-related tokens. Notably, the speaker-score matrix acts as the weight for the corresponding speaker-related token, representing the speaker's identity. The singular value transformation-assisted matrix is generated through the recomposition of the decomposed orthonormal eigenvectors matrix and non-linear transformed singular through Singular Value Decomposition (SVD). This process prevents the degradation of speaker distinctiveness caused by the introduction of other speakers' identity information. By multiplying the singular value transformation-assisted matrix and speaker-related tokens, we generate the anonymized speaker identity representation, thereby producing anonymized speech that is both natural and distinctive. Experiments on VoicePrivacy Challenge datasets demonstrate the effectiveness of our approach in protecting speaker privacy under all attack scenarios while maintaining speech naturalness and distinctiveness.
BibTeX
@article{yao2024distinctive,
  author = {Jixun Yao and Qing Wang and Pengcheng Guo and Ziqian Ning and Lei Xie},
  title = {{Distinctive and Natural Speaker Anonymization via Singular Value Transformation-Assisted Matrix}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2024},
  url = {https://openreview.net/forum?id=dHS9TTno8V},
}
Figure