Characterizing Stereotypical Bias from Privacy-preserving Pre-Training

Stefan Arnold, Rene Gröbner, Annika SchreinerACL 2024

TL;DR

This paper investigates how applying differential privacy (DP) techniques to text data affects stereotypical biases in language models (LMs). They found that while increasing privacy generally reduces bias, this effect isn't uniform across all social domains, highlighting the need for careful bias assessment in privacy-preserving LMs.

What Problem Does This Paper Address?

Large language models (LMs) are trained on vast amounts of text data, which often contain sensitive information and reflect societal biases. Recovering private information from LMs is a growing concern, leading to the development of privacy-preserving techniques like Differential Privacy (DP). DP aims to make model outputs indistinguishable regardless of whether a specific data point was included in the training set.

However, directly applying DP to LMs can be computationally challenging. A common workaround is to perturb the raw text data before training the LM. This approach, known as text privatization, has been shown to degrade LM performance (utility). A key question arises: if text privatization degrades LM capabilities, could it also inadvertently reduce the stereotypical biases often found in LMs? Prior research suggests a correlation between linguistic proficiency and stereotypical bias, leading to the hypothesis that privacy-preserving text privatization might cancel out undesirable biases. This paper aims to empirically test this hypothesis.

Methods

The researchers employed a BERT model for their experiments, leveraging its masked language modeling and next sentence prediction capabilities for pre-training. They used a web-scraped dataset called WebText for training.

To privatize the text, they implemented the "Madlib" mechanism, which perturbs words in a text by substituting them with other words from an embedding space. This method is grounded in metric DP and uses word embeddings (specifically GloVe) to find plausible substitutions within a defined radius, controlled by a privacy budget ($\epsilon$). They tested two levels of privacy: $\epsilon=5$ (high privacy) and $\epsilon=10$ (lower privacy).

To measure stereotypical bias, they utilized two established benchmarks:

  1. StereoSet: This dataset measures bias at both sentence (intrasentence) and discourse (intersentence) levels. It assesses how likely an LM is to associate a social group with a stereotypical, anti-stereotypical, or unrelated attribute.
  2. CrowS-Pairs: This benchmark consists of pairs of minimally different sentences that either reflect a stereotype or violate it. The LM's preference for one sentence over the other indicates its bias.

They compared the bias scores of BERT models trained on privatized text with a control BERT model trained on the original, un-privatized WebText.

Datasets

  • WebText: Used for pre-training the BERT models. It's a large, web-scraped dataset.
  • GloVe embeddings: Used by the Madlib mechanism for text privatization.
  • StereoSet: Used for evaluating stereotypical bias at sentence and discourse levels.
  • CrowS-Pairs: Used for evaluating stereotypical bias across a broader range of social categories.

Key Insights and Ideas

  • Privacy vs. Bias Trade-off is Nuanced: The core finding is that while increasing privacy generally does reduce stereotypical bias, this reduction is not uniform across all social categories. This challenges the simple assumption that privacy-preserving methods automatically lead to fairer models.
  • Intrasentence vs. Intersentence Bias: The study observed that text privatization affects intrasentence (sentence-level) bias more acutely than intersentence (discourse-level) bias. This suggests that masking and predicting words within a sentence is more sensitive to privacy perturbations than understanding broader discourse context.
  • Social Categories React Differently: The impact of privacy on bias varies significantly depending on the social category (e.g., gender, race, religion, profession). Some categories show consistent reduction in bias, while others might even see amplification or stabilization, indicating complex interactions.
  • Importance of Careful Measurement: The findings underscore the critical need for thorough and nuanced bias assessment when deploying privacy-preserving LMs. Simply achieving privacy doesn't guarantee fairness.

Key Results

  • General Bias Reduction: Across both StereoSet and CrowS-Pairs benchmarks, increasing the privacy budget (i.e., tightening privacy, lower $\epsilon$) generally led to a decrease in stereotypical bias scores. For instance, average intrasentence stereotype scores on StereoSet decreased from 0.6054 (no privacy) to 0.5711 ($\epsilon=10$) and 0.5382 ($\epsilon=5$).
  • Varied Impact Across Categories:
    • StereoSet: For intrasentence tasks, gender and race showed a decrease in bias with increased privacy. Religion showed an increase, and profession showed a decrease.
  • CrowS-Pairs: Categories like age and race showed amplification of bias with increased privacy, while occupation and sexuality showed attenuation. Gender and nationality showed fluctuating responses.
  • Utility Degradation: The paper notes that pseudo-perplexity scores (a measure of LM's linguistic proficiency) significantly increased with higher privacy settings, indicating a substantial degradation in language modeling capabilities, consistent with prior work.

Limitations and Future Work

  • WebText Corpus Dependency: The study acknowledges that insights derived from WebText might be influenced by its inherent biases and spurious correlations. Future work should explore diverse datasets.
  • GloVe Embedding Biases: The reliance on GloVe embeddings for text privatization introduces another potential source of bias. Investigating alternative embeddings is suggested.
  • Randomness in Privatization: While the authors reproduced experiments with different random seeds to mitigate concerns about randomness, the inherent probabilistic nature of Madlib could still influence bias patterns across social categories.
  • Future Directions: The authors advocate for careful bias measurement when deploying privacy-preserving LMs. Future research could explore alternative text privatization techniques, investigate the underlying mechanisms driving the differential impact on social categories, and develop methods to mitigate bias while preserving privacy.

Summary updated

BibTeX
@inproceedings{arnold2024characterizing,
  author = {Stefan Arnold and Rene Gröbner and Annika Schreiner},
  title = {{Characterizing Stereotypical Bias from Privacy-preserving Pre-Training}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure