Introduction
Large-scale pre-trained language models (PTLMs) such as BERT and GPT-2 have become the backbone of modern natural language processing. However, their tendency to generate biased or toxic content remains a significant concern. While previous research has focused on static word associations, this paper investigates how models generate toxic content when prompted by benign, everyday contexts. Researchers at HKUST developed a probing framework to determine if the "common sense" reasoning within these models is inherently poisoned by social stereotypes across multiple languages, including English, French, and Arabic.
Methodology
The authors employed a rule-based "cloze" (fill-in-the-blank) methodology based on the ATOMIC knowledge graph. This graph provides thousands of everyday cause-effect relations, which were used to generate templates like: [Social Group] [ACTION] because [PRONOUN] is [MASK].
The probing process involved:
- Template Generation: Creating prompts that pair social groups (covering race, religion, gender, and marginalized status) with neutral actions.
- Model Probing: Using PTLMs to predict the masked token.
- Toxicity Classification: To ensure fair evaluation, a two-step filtering process was used. First, templates flagged as toxic before the model's prediction were removed to account for classifier bias against identity terms. Second, the completed sentences were analyzed to measure toxicity introduced solely by the model.
- Human Validation: Expert annotators verified a sample of predictions to ensure automated labels aligned with human intuition regarding insults and stereotypes.
Results and Discussion
The study found that models frequently inject toxicity into otherwise "clean" scenarios. For example, while a model might attribute "hunger" to a general person making a snack, it might attribute "poverty" to a "Black woman" performing the same action.
Key quantitative findings include:
- Model Variance: BERT demonstrated a significantly higher toxicity rate (14.20% at top-1) compared to RoBERTa (5.95%) and GPT-2 (3.19%).
- Cross-Lingual Nuance: Toxicity patterns varied by language. CamemBERT (French) showed the highest raw toxicity rate at 23.38%. In Arabic, AraBERT frequently used culturally specific animal-based insults, such as "dog" or "donkey," to describe marginalized groups.
- Targeted Vulnerability: Certain groups were disproportionately affected; in BERT, "Refugees" and "Disabled people" saw toxicity rates exceeding 40%.
Conclusion
This research demonstrates that toxicity in large language models is deeply embedded in their predictive reasoning. By using a two-step filtering method, the authors successfully isolated model-generated bias from classifier-induced bias. Although limited to single-token predictions, this framework provides a robust foundation for auditing and eventually mitigating toxicity in future model iterations through techniques like "toxicity anchors" during fine-tuning.
Summary updated