Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervision

Chandra Khatri, Behnam Hedayatnia, Rahul Goel, Anushree Venkatesh, Raefer Gabriel, Arindam MandalarXiv 2018 · OpenReview

Abstract

As open-ended human-chatbot interaction becomes commonplace, sensitive content detection gains importance. In this work, we propose a two stage semi-supervised approach to bootstrap large-scale data for automatic sensitive language detection from publicly available web resources. We explore various data selection methods including 1) using a blacklist to rank online discussion forums by the level of their sensitiveness followed by randomly sampling utterances and 2) training a weakly supervised model in conjunction with the blacklist for scoring sentences from online discussion forums to curate a dataset. Our data collection strategy is flexible and allows the models to detect implicit sensitive content for which manual annotations may be difficult. We train models using publicly available annotated datasets as well as using the proposed large-scale semi-supervised datasets. Results show that a model trained on this collected data outperforms the baseline models by a large margin on both in-domain and out-of-domain testsets, achieving an F1 score of 95.5% on an out-of-domain testset compared to a score of 75% for models trained on public datasets.

1. Introduction

The proliferation of open-domain conversational AI systems, such as Alexa and Google Assistant, has increased the frequency of interactions involving sensitive or offensive user inputs. Detecting such language is challenging because simple blacklists fail to capture implicit abuse, sarcasm, and context-dependent insults that bypass traditional keyword-based filters. Furthermore, existing supervised datasets for hate speech detection are often limited in scale and do not generalize well to the informal, spoken-language style of human-chatbot interactions. This work addresses these challenges by introducing a two-stage semi-supervised pipeline to generate a massive, high-quality training dataset from public web resources.

2. Methodology: Two-Stage Semi-Supervision

The core of our approach is a Two-Stage Semi-Supervised Pipeline designed to bootstrap a large-scale dataset from Reddit, leveraging the platform's community-based structure.

The Two-Stage Semi-Supervised Pipeline for data curation.

2.1 Stage 1: Blacklist-based Subreddit Ranking

We utilize a curated blacklist of over 800 sensitive terms to rank approximately 5,000 subreddits based on their overall sensitivity (the ratio of blacklist terms to total words). By sampling utterances from the most and least sensitive subreddits, we create an initial "Bootstrap" dataset to train a preliminary classifier.

2.2 Stage 2: Weakly Supervised Sampling

In the second stage, a binary classifier is trained on the public Kaggle "Toxic Comments" dataset. This model acts as a weak supervisor, working alongside the blacklist to filter the ranked subreddits. This process identifies 10 million high-confidence sensitive utterances and 10 million high-confidence non-sensitive utterances, resulting in a robust 20-million utterance dataset for final model training.

3. Experimental Setup and Datasets

We evaluate our models across several datasets to measure in-domain performance and out-of-domain generalization.

Comparison of dataset statistics across various sources.
  • Reddit: A 20 million utterance corpus used for the semi-supervised bootstrapping process.
  • Toxic Comments (Wikipedia): Approximately 160k annotated sentences used for weak supervision and in-domain testing.
  • Twitter: A dataset of ~25k annotated tweets used as a baseline.
  • Chatbot Dataset: A manually annotated out-of-domain test set consisting of ~6,000 spoken utterances from a large-scale chatbot competition. This dataset includes multi-class labels such as Inappropriate, Insulting, and Racist.

4. Results and Discussion

Our "Two-Stage (TS) Bootstrap" model demonstrates significant improvements over baseline models trained on smaller, manually annotated datasets.

4.1 Out-of-Domain Generalization

On the Chatbot test set, the TS Bootstrap model achieved an F1 score of 95.5%. In contrast, models trained solely on Twitter or Toxic Comments datasets achieved F1 scores of 74.9% and 74.0%, respectively. This highlights the importance of scale and conversational context provided by the Reddit-bootstrapped data.

4.2 Addressing the Recall Gap

Traditional blacklists are precise but suffer from low recall (24.6% on the Chatbot dataset). The TS Bootstrap model effectively addresses this, achieving 92.0% recall. Furthermore, across seven distinct sensitive sub-classes (e.g., hate speech, pornographic content), the model maintained a weighted average recall of 95.5%, compared to 73.2% for baseline models.

5. Conclusion

This research demonstrates that leveraging community-level topical clustering through semi-supervision can overcome the limitations of manual annotation and static blacklists. By training on a massive, context-rich conversational dataset, we can build robust moderation systems that generalize across diverse sensitive categories and handle the nuances of open-domain human-chatbot interaction.

Summary updated

Abstract

As open-ended human-chatbot interaction becomes commonplace, sensitive content detection gains importance. In this work, we propose a two stage semi-supervised approach to bootstrap large-scale data for automatic sensitive language detection from publicly available web resources. We explore various data selection methods including 1) using a blacklist to rank online discussion forums by the level of their sensitiveness followed by randomly sampling utterances and 2) training a weakly supervised model in conjunction with the blacklist for scoring sentences from online discussion forums to curate a dataset. Our data collection strategy is flexible and allows the models to detect implicit sensitive content for which manual annotations may be difficult. We train models using publicly available annotated datasets as well as using the proposed large-scale semi-supervised datasets. We evaluate the performance of all the models on Twitter and Toxic Wikipedia comments testsets as well as on a manually annotated spoken language dataset collected during a large scale chatbot competition. Results show that a model trained on this collected data outperforms the baseline models by a large margin on both in-domain and out-of-domain testsets, achieving an F1 score of 95.5% on an out-of-domain testset compared to a score of 75% for models trained on public datasets. We also showcase that large scale two stage semi-supervision generalizes well across multiple classes of sensitivities such as hate speech, racism, sexual and pornographic content, etc. without even providing explicit labels for these classes, leading to an average recall of 95.5% versus the models trained using annotated public datasets which achieve an average recall of 73.2% across seven sensitive classes on out-of-domain testsets.
BibTeX
@misc{khatri2018detecting,
  author = {Chandra Khatri and Behnam Hedayatnia and Rahul Goel and Anushree Venkatesh and Raefer Gabriel and Arindam Mandal},
  title = {{Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervision}},
  year = {2018},
  eprint = {1811.12900},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.12900},
}
Figure