Probing Toxic Content in Large Pre-Trained Language Models

Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, Dit-Yan Yeung

Introduction

Large-scale pre-trained language models (PTLMs) such as BERT and GPT-2 have become the backbone of modern natural language processing. However, their tendency to generate biased or toxic content remains a significant concern. While previous research has focused on static word associations, this paper investigates how models generate toxic content when prompted by benign, everyday contexts. Researchers at HKUST developed a probing framework to determine if the "common sense" reasoning within these models is inherently poisoned by social stereotypes across multiple languages, including English, French, and Arabic.

Methodology

The authors employed a rule-based "cloze" (fill-in-the-blank) methodology based on the ATOMIC knowledge graph. This graph provides thousands of everyday cause-effect relations, which were used to generate templates like: [Social Group] [ACTION] because [PRONOUN] is [MASK].

Overview of the Masked LM probing framework, showing how a benign sentence is used to elicit potentially toxic predictions from a model.

The probing process involved:

  1. Template Generation: Creating prompts that pair social groups (covering race, religion, gender, and marginalized status) with neutral actions.
  2. Model Probing: Using PTLMs to predict the masked token.
  3. Toxicity Classification: To ensure fair evaluation, a two-step filtering process was used. First, templates flagged as toxic before the model's prediction were removed to account for classifier bias against identity terms. Second, the completed sentences were analyzed to measure toxicity introduced solely by the model.
  4. Human Validation: Expert annotators verified a sample of predictions to ensure automated labels aligned with human intuition regarding insults and stereotypes.

Results and Discussion

The study found that models frequently inject toxicity into otherwise "clean" scenarios. For example, while a model might attribute "hunger" to a general person making a snack, it might attribute "poverty" to a "Black woman" performing the same action.

Quantitative comparison of toxicity rates across various pre-trained language models at different prediction depths.

Key quantitative findings include:

  • Model Variance: BERT demonstrated a significantly higher toxicity rate (14.20% at top-1) compared to RoBERTa (5.95%) and GPT-2 (3.19%).
  • Cross-Lingual Nuance: Toxicity patterns varied by language. CamemBERT (French) showed the highest raw toxicity rate at 23.38%. In Arabic, AraBERT frequently used culturally specific animal-based insults, such as "dog" or "donkey," to describe marginalized groups.
  • Targeted Vulnerability: Certain groups were disproportionately affected; in BERT, "Refugees" and "Disabled people" saw toxicity rates exceeding 40%.

Conclusion

This research demonstrates that toxicity in large language models is deeply embedded in their predictive reasoning. By using a two-step filtering method, the authors successfully isolated model-generated bias from classifier-induced bias. Although limited to single-token predictions, this framework provides a robust foundation for auditing and eventually mitigating toxicity in future model iterations through techniques like "toxicity anchors" during fine-tuning.

Summary updated

BibTeX
@inproceedings{ousidhoum2021probing,
  author = {Nedjma Ousidhoum and Xinran Zhao and Tianqing Fang and Yangqiu Song and Dit-Yan Yeung},
  title = {{Probing Toxic Content in Large Pre-Trained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.329},
}
Figure