Perturbation Sensitivity Analysis to Detect Unintended Model Biases

Vinodkumar Prabhakaran, Ben Hutchinson, Margaret Mitchell

TL;DR

This paper introduces Perturbation Sensitivity Analysis (PSA), a novel method to detect unintended biases in NLP models by systematically swapping named entities within sentences. PSA reveals how sensitive models are to these changes, offering a generalizable way to identify biases without requiring new annotated datasets.

What Problem Does This Paper Address?

Modern NLP models, trained on vast amounts of text data, often inherit and even amplify societal biases present in that data. This can lead to unfair outcomes, particularly when these models are used in sensitive applications like content moderation. While existing fairness research often focuses on specific demographic groups or requires extensive manual annotation, this paper addresses the need for a more general and accessible method to detect biases related to any named entities (people, places, etc.) within existing text. The core problem is that NLP models might associate certain sentiments or toxicity levels with specific individuals or groups simply due to incidental co-occurrences in training data, rather than genuine semantic meaning.

Methods

The paper proposes Perturbation Sensitivity Analysis (PSA), a framework designed to measure how much an NLP model's output changes when a named entity in a sentence is replaced with another of the same type.

Here's how it works:

  1. Sentence Retrieval: They first collect a set of sentences from a target corpus that contain referring expressions (like pronouns or proper names) for a specific entity type (e.g., people).
  2. Anchor Selection: A referring expression in each sentence is chosen as an "anchor."
  3. Perturbation: The anchor is systematically replaced with various named entities from a predefined list (e.g., a list of famous people).
  4. Sensitivity Measurement: The model is run on these perturbed sentences, and its output scores are compared to the original sentence's score.

They define three key metrics for measuring this sensitivity:

  • Perturbation Score Sensitivity (ScoreSens): The average difference between the perturbed score and the original score.
  • Perturbation Score Deviation (ScoreDev): The standard deviation of scores across different perturbations for a given sentence.
  • Perturbation Score Range (ScoreRange): The difference between the maximum and minimum scores across all perturbations for a given sentence.

For binary classifiers, they also introduce Perturbation Label Distance (LabelDist), which measures how often the model's predicted label flips (e.g., from toxic to non-toxic) due to name perturbation.

Datasets

The paper uses four publicly available corpora of online comments:

  • Facebook comments on politicians' posts (FB-Pol.)
  • Facebook comments on public figures' posts (FB-Pub.)
  • Reddit comments
  • Fitocracy forums

These corpora were selected for their social and topical diversity. For evaluation, they extracted 1000 comments per corpus, focusing on those with third-person singular pronouns as anchors and a maximum length of 50 words. They did not introduce new annotated benchmarks but rather applied their analysis to existing text.

Key Insights and Ideas

  • Universality of Bias: The core idea is that models should be insensitive to the identity of named entities when the underlying semantic meaning of the sentence remains the same. For example, "I hate Katy Perry" and "I hate Taylor Swift" should ideally have similar toxicity scores.
  • Pronouns as Anchors: Using pronouns (like "he" or "she") as anchors for perturbation is a clever way to create a generalizable framework. Pronouns are common across languages and less likely to carry inherent semantic bias themselves, allowing for a cleaner assessment of how names influence model predictions.
  • Beyond Specific Groups: PSA offers a way to detect biases related to any named entity, not just pre-defined sensitive groups (like race or gender). This makes it a more flexible tool for uncovering a wider range of potential harms.
  • Downstream Impact: The paper demonstrates that these score changes can lead to actual label flips (e.g., classifying a comment as toxic or not toxic), highlighting the practical implications of this sensitivity.

Key Results

  • Significant Sensitivity: Both toxicity and sentiment models showed significant sensitivity to name perturbations across all four corpora.
  • Sentiment Models More Sensitive: The sentiment analysis model exhibited higher ScoreDev and ScoreRange values compared to the toxicity model, suggesting it is more susceptible to biases introduced by named entities.
  • Career Type Matters: For the toxicity model on Reddit, names associated with "leaders" (politicians) and "actors" showed higher toxicity associations than those of "musicians" and "athletes."
  • Threshold Dependence: While the magnitude of score change (|f(xn) - f(x)|) was negatively correlated with the original score (meaning larger changes happened on less extreme scores), the label flip (LabelDist) was still significant even at high thresholds, indicating that perturbation can flip classifications even for strongly-scored examples. LabelDist values of 0.10-0.40 suggest that 10-40% of sentences could flip labels due to name perturbation.

Limitations and Future Work

  • Assumption of Semantic Equivalence: The core assumption of PSA is that replacing one named entity with another of the same type should not change the model's output. However, the authors acknowledge that in some cases, proper names carry pragmatic meanings (e.g., "he is like Hitler" vs. "he is like Gandhi") that are semantically relevant and should lead to different scores. Careful corpus construction is needed to mitigate this.
  • Corpus Construction: The effectiveness of PSA relies on the quality and diversity of the input corpus (X) and the list of names (N). Biases in these lists could confound results.
  • Future Work: The authors suggest extending PSA to other entity types (locations, organizations), other NLP models (beyond classification), and different languages. They also propose adapting the method for bias mitigation, either by perturbing names in training data or by using post-processing techniques to average scores across perturbed sentences.

Summary updated

Abstract

Data-driven statistical Natural Language Processing (NLP) techniques leverage large amounts of language data to build models that can understand language. However, most language data reflect the public discourse at the time the data was produced, and hence NLP models are susceptible to learning incidental associations around named referents at a particular point in time, in addition to general linguistic meaning. An NLP system designed to model notions such as sentiment and toxicity should ideally produce scores that are independent of the identity of such entities mentioned in text and their social associations. For example, in a general purpose sentiment analysis system, a phrase such as I hate Katy Perry should be interpreted as having the same sentiment as I hate Taylor Swift. Based on this idea, we propose a generic evaluation framework, Perturbation Sensitivity Analysis, which detects unintended model biases related to named entities, and requires no new annotations or corpora. We demonstrate the utility of this analysis by employing it on two different NLP models --- a sentiment model and a toxicity model --- applied on online comments in English language from four different genres.
BibTeX
@inproceedings{prabhakaran2019perturbation,
  author = {Vinodkumar Prabhakaran and Ben Hutchinson and Margaret Mitchell},
  title = {{Perturbation Sensitivity Analysis to Detect Unintended Model Biases}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1578},
}
Figure