Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP

Timo Schick, Sahana Udupa, Hinrich Schütze

TL;DR

Large Language Models (LLMs) are surprisingly good at identifying their own toxic outputs—a capability the authors call "self-diagnosis." By leveraging this internal knowledge, the researchers developed a "self-debiasing" decoding algorithm that reduces the probability of generating biased or harmful text in real-time, without requiring any retraining or manual word blacklists.

What Problem Does This Paper Address?

Modern LLMs are trained on massive, unfiltered crawls of the internet. Consequently, they "inhale" and subsequently "exhale" the internet's worst traits: racism, sexism, and various forms of social bias.

Existing solutions generally fall into two camps, both with significant flaws:

  1. Manual Word Lists: Curation is labor-intensive, and banning words like "sex" or "gay" can prevent models from discussing important topics or even hurt their general performance.
  2. Retraining/Fine-tuning: Training a model on "clean" data is computationally expensive and requires massive amounts of high-quality, human-annotated datasets which are hard to come by for every possible bias.

The authors ask: Can we make a model "police" itself using the knowledge it already acquired during pretraining?

Methods

The paper introduces two core concepts:

1. Self-Diagnosis

The researchers found that if you show a model a piece of text and ask it a simple question—"Does the above text contain [bias type]?"—large models can answer with high accuracy. They use a template-based approach where they compare the model's probability of generating "Yes" vs. "No" for a given attribute (like "sexist" or "violent").

2. Self-Debiasing

This is a novel decoding algorithm. During text generation, for every next word the model considers, the algorithm performs a "shadow" calculation. It asks: How likely is this word if I explicitly encourage the model to be biased?

Specifically, it calculates the difference ($\Delta$) between:

  • The probability of a word in the original context.
  • The probability of that same word when the model is prompted to be biased (e.g., "The following text is racist: [input]").

If a word becomes significantly more likely in the biased prompt, the algorithm applies a penalty to that word's probability in the final output. This "soft" filtering happens token-by-token during the generation process.

Datasets

The authors evaluated their approach using several established benchmarks:

  • RealToxicityPrompts: A dataset of thousands of naturally occurring prompts that often trigger toxic continuations in LLMs.
  • CrowS-Pairs: A benchmark for measuring nine types of social bias (e.g., race, religion, age) in masked language models like BERT and RoBERTa.
  • Wikitext-2: Used to ensure that debiasing doesn't destroy the model's general language capabilities (measuring "perplexity").
  • Perspective API: An automated tool used to provide "silver-standard" labels for toxicity during evaluation.

Key Insights and Ideas

  • The "Scale" Threshold: Self-diagnosis is an emergent property of scale. Small models (like GPT-2 small) are essentially guessing, but as models grow (GPT2-XL, T5-XXL), their ability to recognize bias improves dramatically.
  • Models "Know" Better: The most striking finding is that models often "know" they are being toxic even while they are being toxic. We don't need to add new knowledge to the model; we just need to change how we extract it.
  • Non-Invasive Surgery: Unlike fine-tuning, self-debiasing requires zero changes to the model's weights. It is a "plug-and-play" algorithm for the decoding step.
  • Complementary Nature: Self-debiasing works well in tandem with other methods like domain-adaptive pretraining (DAPT), providing an extra layer of safety.

Key Results

  • Toxicity Reduction: For GPT2-XL, self-debiasing reduced the probability of generating toxic content by over 50% on average across six different attributes.
  • Human Validation: In a blind study, human evaluators confirmed that self-debiasing significantly reduced toxicity. Interestingly, they also rated the debiased outputs as slightly more fluent and coherent than the original biased ones.
  • Social Bias: On the CrowS-Pairs dataset, the method reduced bias scores by up to 6.7 percentage points, showing it works for subtle social stereotypes, not just overt toxicity.
  • Minimal Performance Hit: The "cost" of this safety—measured by how much it confuses the model on standard text (perplexity)—was found to be very low, especially for moderate debiasing strengths.

Limitations and Future Work

  • Computational Overhead: Because the model has to process a "debiasing prompt" for every attribute you want to filter, the decoding time increases linearly with the number of attributes.
  • Greedy Logic: The algorithm makes decisions word-by-word. It might occasionally penalize a word that seems bad in isolation but is actually part of a harmless sentence (though beam search can mitigate this).
  • English-Only: The study was limited to English models and datasets.
  • The "Definition" Problem: The method relies on the model's internal understanding of words like "racist" or "sexist." If the model's internal definition is flawed or limited, the debiasing will be too. The authors suggest future work should involve "people-centric" processes to define these attributes more robustly.

Summary updated

Abstract

This paper contains prompts and model outputs that are offensive in nature.When trained on large, unfiltered crawls from the Internet, language models pick up and reproduce all kinds of undesirable biases that can be found in the data: They often generate racist, sexist, violent, or otherwise toxic language. As large models require millions of training examples to achieve good performance, it is difficult to completely prevent them from being exposed to such content. In this paper, we first demonstrate a surprising finding: Pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce. We refer to this capability as self-diagnosis. Based on this finding, we then propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. We refer to this approach as self-debiasing. Self-debiasing does not rely on manually curated word lists, nor does it require any training data or changes to the model’s parameters. While we by no means eliminate the issue of language models generating biased text, we believe our approach to be an important step in this direction.1
BibTeX
@article{schick2021self,
  author = {Timo Schick and Sahana Udupa and Hinrich Schütze},
  title = {{Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=09S28ETcBI},
}

Loading related content...

Figure