TL;DR
This paper introduces a novel method using a Vector-Quantized Variational Autoencoder (VQ-VAE) combined with adversarial learning and mutual information loss to remove or manipulate sensitive attributes like age and sex from speaker representations. This approach aims to enhance privacy without significantly degrading the utility of these representations for tasks like speaker verification.
What Problem Does This Paper Address?
Speaker representations, or embeddings, are powerful tools in speech technology, enabling tasks from speaker recognition to voice anonymization. However, these embeddings can inadvertently encode sensitive personal information, such as age, sex, and even health status. This raises significant privacy concerns, especially when these representations are processed remotely or shared. Current methods for addressing this often involve complex cryptographic protocols with high computational costs or simpler privacy-oriented manipulation techniques that may lack formal privacy guarantees. This paper tackles the challenge of effectively removing or manipulating specific sensitive attributes from speaker embeddings while preserving their usefulness for downstream tasks, particularly Automatic Speaker Verification (ASV).
Methods
The core of the proposed method is a Vector-Quantized Variational Autoencoder (VQ-VAE). VQ-VAEs are known for their ability to learn structured and disentangled latent representations, making them suitable for manipulating specific information. The authors enhance this VQ-VAE with two key components:
- Adversarial Classifier: An external classifier is trained to predict a sensitive attribute (e.g., sex) from the VQ-VAE's latent representation. This classifier is then used in an adversarial setup with the VQ-VAE. The VQ-VAE is trained to "fool" this classifier, effectively learning to remove the attribute information from its latent space. A Gradient Reversal Layer (GRL) is used to achieve this adversarial training.
- Mutual Information (MI) Loss: To complement the adversarial approach and address potential weaknesses (like adversarial classifiers failing to generalize), the authors incorporate a non-parametric mutual information loss. This loss aims to minimize the information shared between the latent representation and the target attribute, promoting representations that are truly invariant to that attribute. They use two specific MI estimators: one for discrete-continuous variables (sex) and another for continuous-continuous variables (age).
The VQ-VAE is trained with a combination of losses: reconstruction loss, codebook diversity loss, an additive angular margin loss for speaker identification (to maintain utility), the adversarial loss, and the mutual information loss.
Datasets
The experiments utilize several datasets:
- VoxCeleb [85]: The primary source for speaker recognition data, used for training speaker embedding extractors and for various partitions in the experiments.
- LibriTTS [86]: Used for cross-domain experiments to assess the model's generalization capabilities to different speech styles (read speech vs. spontaneous interviews).
- AgeVoxCeleb [87]: An age-annotated partition of VoxCeleb, used for age-related experiments.
- VoxCelebPT [88]: A Portuguese version of VoxCeleb, used as held-out test data for age experiments.
No new benchmarks were introduced; existing datasets were partitioned and utilized for specific training and testing purposes.
Key Insights and Ideas
- Synergy of Adversarial and MI Losses: The paper demonstrates that combining adversarial training with mutual information loss provides a more robust approach to attribute removal than either method alone. The adversarial component pushes for attribute removal, while the MI loss ensures that the learned representations are genuinely invariant, not just fooling a specific classifier.
- Attribute Manipulation is Feasible: Beyond just removal, the method can effectively manipulate attributes by feeding external information to the decoder. This opens doors for controlling speaker characteristics in synthesized speech or for specific privacy-preserving applications.
- Trade-off Between Privacy and Utility: The study highlights the inherent trade-off. While privacy improves with the proposed components, there's a slight degradation in speaker verification performance. The authors show that their combined approach achieves a good balance.
- Cross-Domain Generalization: The model shows promising results in cross-domain settings, indicating that privacy-preserving techniques can be adapted to different types of speech data, though domain-specific training still yields the best performance.
Key Results
- Privacy: For sex information, the combined VQ-VAE + ADV + MI model achieved sex classification performance close to chance level (UAR around 50.91% for informed attackers), significantly outperforming the baseline VQ-VAE and even the NFzLLR method in some privacy metrics. For age, similar privacy improvements were observed, with correlation coefficients dropping significantly.
- Utility: Speaker verification performance (EER) showed only a slight degradation: 0.6% for the combined sex removal model compared to the original data, and 3.4% for the age removal model. This indicates that the utility of the representations is largely preserved.
- Manipulation: The models successfully manipulated attribute information, achieving high "fake" attribute prediction scores (e.g., 90% UAR for sex, 0.91 CCC for age) with pre-trained classifiers, demonstrating the ability to replace true attributes with external ones.
Limitations and Future Work
- Incomplete Removal: The authors acknowledge that their method cannot entirely remove attribute information, especially against stronger, informed attackers. Privacy metrics are close to chance but not zero.
- Utility Measurement: Utility is primarily measured by ASV performance. The authors suggest evaluating effects on other downstream tasks (e.g., emotion recognition, speech synthesis) for a more comprehensive understanding.
- Trade-off Control: Explicitly controlling the trade-off between privacy and utility remains a challenge. While weights can be adjusted, the relationship is not linear, and further research is needed to make this control more intuitive.
- Future Work: The authors suggest exploring multi-class attributes (like accent), paralinguistic traits (emotion), domain generalization, cross-attribute effects, and applying the model to voice conversion and text-to-speech tasks. They also highlight the importance of informing users about the privacy-utility trade-offs.
Summary updated