Fortifying Toxic Speech Detectors Against Veiled Toxicity

Xiaochuang Han, Yulia Tsvetkov

Abstract

Modern toxic speech detectors are incompetent in recognizing disguised offensive language, such as adversarial attacks that deliberately avoid known toxic lexicons, or manifestations of implicit bias. Building a large annotated dataset for such veiled toxicity can be very expensive. In this work, we propose a framework aimed at fortifying existing toxic speech detectors without a large labeled corpus of veiled toxicity. Just a handful of probing examples are used to surface orders of magnitude more disguised offenses. We augment the toxic speech detector's training data with these discovered offensive examples, thereby making it more robust to veiled toxicity while preserving its utility in detecting overt toxicity.

Introduction

While current toxicity filters, such as Google’s Perspective API, effectively identify overt toxicity—including profanity and blatant slurs—they struggle significantly with veiled toxicity. This category encompasses adversarial attacks designed to bypass filters, implicit biases like microaggressions, and the use of innocent-sounding codewords to target specific groups.

The traditional approach of building massive, hand-annotated datasets for these subtle cases is prohibitively expensive. Random sampling often fails to capture enough relevant examples, and human annotators frequently disagree on what constitutes a "subtle" offense. This paper introduces a framework to "fortify" these detectors using only a small set of labeled probing examples.

Methodology: Surface Hidden Offenses

The researchers developed a "student-teacher" framework that operates without needing the original model's internal weights.

  1. Student Model: A BERT-based "student" model is trained to mimic a "compromised" classifier (the "teacher," e.g., Perspective API).
  2. Probing: A small set (20–100) of known veiled offensive examples, termed probing examples, is introduced.
  3. Influence Tracking: The core of the system uses interpretable machine learning to identify which training examples most influenced the model to misclassify probing examples as non-toxic. Among various methods, the Gradient Product (TrackIn) proved most effective.
Distribution of training example influence ranks across different toxicity classes.

As shown in Figure 1, veiled offenses tend to have a distinct influence profile compared to clean messages or overt offenses, allowing them to be surfaced from "non-toxic" datasets.

  1. Data Augmentation: Once influential (likely mislabeled) examples are identified, the system either fixes their labels using gold-standard annotations or "flips" the labels to "toxic" automatically.

Key Results and Insights

The framework reveals that existing "non-toxic" training sets are often "polluted" with disguised offenses. By using influence as a search tool, the researchers found a "needle in the haystack" without needing massive new data collection.

Performance metrics for various fortification methods and operations.

The results, summarized in Figure 3, demonstrate a dramatic improvement in detection capabilities:

  • Massive Recall Boost: The original model's recall on veiled toxicity (VO) was a mere 1.2%. After identifying and flipping the labels of the top 2,000 influential examples using the Gradient Product method, recall jumped to 51.1%.
  • Efficiency: Probing with just 100 examples surfaced nearly 1,000 additional veiled offenses, representing a 10x return on annotation effort.
  • Preserving Utility: The model maintained high performance (over 99%) in detecting overt toxicity (OO), ensuring basic functionality remained intact.

Discussion and Limitations

While "label flipping" without human verification significantly boosted recall, it introduced a precision-recall trade-off. Accuracy on truly non-toxic "clean" data (NO) dropped from 99.6% to 87.6%, indicating the model became somewhat over-sensitive.

Furthermore, the surfaced messages did not always align with the specific social groups of the probing examples. Future work should focus on making detectors more group-aware and improving their understanding of conversational context and underlying presuppositions to better distinguish subtle toxicity from benign speech.

Summary updated

Abstract

Modern toxic speech detectors are incompetent in recognizing disguised offensive language, such as adversarial attacks that deliberately avoid known toxic lexicons, or manifestations of implicit bias. Building a large annotated dataset for such veiled toxicity can be very expensive. In this work, we propose a framework aimed at fortifying existing toxic speech detectors without a large labeled corpus of veiled toxicity. Just a handful of probing examples are used to surface orders of magnitude more disguised offenses. We augment the toxic speech detector's training data with these discovered offensive examples, thereby making it more robust to veiled toxicity while preserving its utility in detecting overt toxicity.
BibTeX
@inproceedings{han2020fortifying,
  author = {Xiaochuang Han and Yulia Tsvetkov},
  title = {{Fortifying Toxic Speech Detectors Against Veiled Toxicity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.622},
}
Figure