Understanding the Tradeoffs in Client-Side Privacy for Speech Recognition

Peter Wu, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency

TL;DR

This paper introduces the concept of "client-side privacy" for speech data, where sensitive information is removed directly on the user's device before being sent to the cloud. The authors explore three methods (signal processing, disentangled representation learning, and adversarial training) for masking gender and accent, finding that while each has strengths, none perfectly balance privacy, performance, and computational efficiency for downstream tasks like ASR.

What Problem Does This Paper Address?

As we increasingly rely on cloud-based services for tasks like automatic speech recognition (ASR), our raw speech data is often uploaded to remote servers. This raises significant privacy concerns, as this data can contain sensitive biometric information such as gender, age, race, and accent. While existing solutions might offer server-side privacy or focus solely on hiding speaker identity, they don't empower users with direct control over what personal attributes are shared. The core problem this paper addresses is the need for client-side privacy – a method that allows users to remove sensitive personal attributes from their speech data on their own device before it's processed by cloud services, ensuring the remaining data is still useful for downstream tasks like ASR. This is particularly challenging due to the need for direct data manipulation, compatibility with various server-side models, and strict efficiency constraints for mobile devices.

Methods

The paper investigates three distinct client-side transform approaches for speech privacy, all aiming to anonymize sensitive attributes while preserving transcribable content for ASR:

  • Pitch Standardization: This method uses signal processing techniques to shift the average pitch of an utterance to a predefined value while preserving formants. It's computationally efficient as it doesn't rely on neural networks.
  • Disentangled Representation Learning: This approach utilizes Variational Autoencoders (VAEs) to learn latent representations of speech. It aims to disentangle speaker-dependent (private) factors from speaker-independent (content) factors. The goal is to encode the audio such that the private factors are excluded during reconstruction, thus masking sensitive attributes.
  • Adversarial Training: This method employs a Generative Adversarial Network (GAN) framework. It involves disentangling input audio into private and content factors, then using a GAN to generate realistic audio. The generator is conditioned on content factors and a new speaker/gender/accent label, while a discriminator attempts to distinguish real from generated audio and also classifies the private attribute. This approach is designed to ensure high-fidelity generation of encrypted audio.

Datasets

The experiments were conducted using the following datasets:

  • VCTK Corpus: Used for training and evaluating privacy attribute classifiers (gender and accent) and for ASR experiments with gender-encrypted audio.
  • LibriSpeech: Used for training the ASR models and for evaluating ASR performance on gender-encrypted audio.

The paper also mentions using a pre-trained ESPNet Transformer ASR model trained on 960 hours of LibriSpeech data. For privacy attribute classification, a modified VGGVox model was used.

Key Insights and Ideas

  • Client-side transforms are key: The paper argues that "client-side transforms" are a promising avenue for achieving client-side privacy, distinct from other privacy algorithms like differential privacy or federated learning.
  • No silver bullet: None of the investigated methods perfectly balance privacy, performance (low ASR error), and computational efficiency. Each method has its own trade-offs.
  • Pitch standardization is efficient but weak on privacy: While computationally cheap, pitch standardization alone is not very effective at masking gender, as classifiers can still identify patterns related to the pitch shift.
  • Neural methods offer better privacy but at a cost: Disentangled representation learning and adversarial training provide better privacy guarantees, but they are more computationally intensive and can sometimes lead to higher ASR error rates.
  • Gender vs. Accent Privacy: Masking gender appears to require a larger data distribution shift than masking accent, leading to better ASR performance after accent encryption compared to gender encryption.
  • Human perception matters: While disentanglement and adversarial methods might perform well on automatic metrics, human listening tests (MOS) revealed that listeners sometimes preferred the adversarial approach over disentanglement, suggesting that disentangled audio can sound robotic even if understandable for ASR.

Key Results

  • Gender Classification: All three proposed approaches could successfully mask gender when no fine-tuning data was available for the classifier. However, with fine-tuning data, the disentanglement approach performed best for higher values of n (number of speakers used for fine-tuning), followed by the adversarial approach with a modified gender loss.
  • ASR Performance (Gender): The signal processing method generally performed best for ASR after gender encryption, likely due to artifacts introduced by neural methods. The adversarial approach with gender-based loss outperformed the speaker-based loss approach.
  • ASR Performance (Accent): ASR results after accent encryption were consistently better than those for gender encryption, indicating that accent transformation requires a smaller data distribution shift.
  • Trade-off revealed: Table I clearly shows a negative correlation between classification accuracy (privacy) and ASR performance (fidelity) when adding Gaussian noise, illustrating the inherent privacy-fidelity tradeoff.
  • MOS Scores: The signal processing approach performed best for female speakers, while the adversarial approach was better for male speakers, suggesting it's more robust.

Limitations and Future Work

  • Limitations:
    • None of the methods achieve an ideal balance of privacy, performance, and efficiency.
  • The ASR models might not be robust enough to the converted samples, or the converted samples might act as adversarial samples during training.
  • Neural methods have significant memory consumption compared to signal processing.
  • Future Work:
    • Develop new models that better reconcile the privacy benefits of neural methods with the performance and memory advantages of signal processing.
  • Analyze other downstream speech tasks, including speech translation and different speech recognition settings.
  • Further research into client-side privacy to ensure safer deployment of cloud-based speech processing services.

Summary updated

Abstract

As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private. Existing solutions rely either on server-side methods or focus on hiding speaker identity. While these approaches reduce certain security concerns, they do not give users client-side control over whether their biometric information is sent to the server. In this paper, we formally define client-side privacy and discuss its three unique technical challenges: (1) direct manipulation of raw data on client devices, (2) adaptability with a broad range of server-side processing models, and (3) low time and space complexity for compatibility with limited-bandwidth devices. Solving these challenges requires new models that achieve high-fidelity reconstruction, privacy preservation of sensitive personal attributes, and efficiency during training and inference. As a step towards client-side privacy for speech recognition, we investigate three techniques spanning signal processing, disentangled representation learning, and adversarial training. Through a series of gender and accent masking tasks, we observe that each method has its unique strengths, but none manage to effectively balance the trade-offs between performance, privacy, and complexity. These insights call for more research in client-side privacy to ensure a safer deployment of cloud-based speech processing services.
BibTeX
@inproceedings{wu2021understanding,
  author = {Peter Wu and Paul Pu Liang and Ruslan Salakhutdinov and Louis-Philippe Morency},
  title = {{Understanding the Tradeoffs in Client-Side Privacy for Speech Recognition}},
  booktitle = {APSIPA ASC},
  year = {2021},
  url = {https://openreview.net/forum?id=BpfWLpSwpbc},
}
Figure