Privacy-Preserving Siamese Feature Extraction for Gender Recognition versus Speaker Identification

Alexandru Nelus, Silas Rech, Timm Koppelmann, Henrik Biermann, Rainer MartinInterspeech 2019

TL;DR

This paper introduces a novel Siamese network-based feature extraction method to protect user privacy in audio applications. By training the network to minimize differences between speakers of the same gender while maximizing differences between genders, it significantly reduces the risk of an attacker identifying a specific speaker from gender recognition features, achieving high gender recognition accuracy with minimal privacy compromise.

What Problem Does This Paper Address?

In today's data-driven world, machine learning (ML) models are increasingly used in everyday devices and services. While powerful, these models also raise significant privacy concerns. This paper focuses on a specific challenge: when an ML model is trained to perform one task, like recognizing a speaker's gender, the extracted features might inadvertently contain enough information for an attacker to perform a more privacy-invasive task, such as identifying the individual speaker. This is particularly relevant given the growing importance of gender equality initiatives that rely on gender recognition, and the increasing emphasis on privacy-by-design principles like the GDPR. The authors aim to create a feature representation that is excellent for gender recognition but deliberately obscures speaker identity.

Methods

The core of the proposed solution is a Siamese network architecture. Traditionally, Siamese networks are used to learn similarity between inputs. Here, the authors adapt this concept for privacy preservation. They train a deep neural network (DNN) feature extractor, denoted as f, using a Siamese training procedure.

The key idea is to feed pairs of audio features into the network. If the two audio samples belong to speakers of the same gender, the network is trained to minimize the Euclidean distance between their extracted feature representations (z1 and z2). Conversely, if the samples belong to speakers of different genders, the network is trained to maximize this distance. This is achieved by minimizing a contrastive loss function that penalizes large distances for same-gender pairs and small distances for different-gender pairs.

The feature extractor f is composed of several convolutional neural network (CNN) groups. After training f in this privacy-preserving manner, it's used in conjunction with a separate gender recognizer network (g). Crucially, during gender recognition, the weights of f are frozen, and only g is trained.

For the attack scenario, a speaker identifier network (a) intercepts the features extracted by f. This attacker network is trained to identify speakers using these features, while f's weights remain frozen.

Datasets

The primary dataset used is the TIMIT corpus. This corpus contains audio data from 420 speakers, with approximately 31 seconds of audio per speaker. The data is divided into:

  • Gender-training and gender-evaluation sets: Used for training and evaluating the gender recognition task.
  • Attacker-training and attacker-evaluation subsets: Used for training and evaluating the speaker identification attack.
  • Gender-testing set: Used for final evaluation of gender recognition.
  • Attacker-testing set: Used for final evaluation of the speaker identification attack.

The paper details a specific split for these sets, with 20 speakers chosen for a cross-validation setup.

Key Insights and Ideas

  • Siamese networks as a privacy tool: The paper demonstrates that the inherent ability of Siamese networks to learn similarity/dissimilarity can be repurposed to create privacy-preserving features. By forcing same-gender speakers into close proximity in the feature space, speaker identity is obscured.
  • Conflicting goals can be managed: The work tackles the challenging scenario of optimizing for two competing goals: accurate gender recognition and robust speaker anonymization.
  • Feature extractor specialization matters: The experiments show that increasing the number of CNN groups (G) in the feature extractor leads to better performance, suggesting that deeper, more specialized feature extraction can enhance both privacy and utility.
  • White-box attack scenario: The authors explicitly define their threat model as a white-box attack, where the attacker knows the algorithm of the trust model, making the privacy challenge more significant.

Key Results

  • Significant privacy improvement: The proposed Siamese feature extraction method reduces attacker accuracy from 65% (in a non-privacy-preserving baseline) to 33%.
  • Minimal impact on gender recognition: The gender recognition accuracy remains high, achieving 94% with the best configuration (G=5), only slightly lower than the non-privacy-preserving baseline of 95%.
  • Comparison to prior work: The Siamese approach outperforms previous privacy-preserving methods like adversarial and variational-information feature extraction, achieving a better trade-off between gender recognition accuracy and attacker accuracy.
  • Effect of network depth: Increasing the number of CNN groups (G) from 1 to 5 improved performance, with G=5 yielding the best results.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the attacker accuracy is significantly reduced, it is not completely eliminated (33% vs. 10% random guess). The authors also note that the "gender reference" and "attacker reference" accuracies were obtained using a single CNN group (G=1) in the non-Siamese baseline, which might not represent the absolute best possible non-privacy-preserving performance.
  • Future Work: The authors intend to expand their investigation to multi-class application scenarios, moving beyond binary gender recognition. They also aim to explore further ways to enhance the privacy guarantees of their system.

Summary updated

BibTeX
@inproceedings{nelus2019privacy,
  author = {Alexandru Nelus and Silas Rech and Timm Koppelmann and Henrik Biermann and Rainer Martin},
  title = {{Privacy-Preserving Siamese Feature Extraction for Gender Recognition versus Speaker Identification}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure