Abstract
As open-ended human-chatbot interaction becomes commonplace, sensitive content detection gains importance. In this work, we propose a two stage semi-supervised approach to bootstrap large-scale data for automatic sensitive language detection from publicly available web resources. We explore various data selection methods including 1) using a blacklist to rank online discussion forums by the level of their sensitiveness followed by randomly sampling utterances and 2) training a weakly supervised model in conjunction with the blacklist for scoring sentences from online discussion forums to curate a dataset. Our data collection strategy is flexible and allows the models to detect implicit sensitive content for which manual annotations may be difficult. We train models using publicly available annotated datasets as well as using the proposed large-scale semi-supervised datasets. Results show that a model trained on this collected data outperforms the baseline models by a large margin on both in-domain and out-of-domain testsets, achieving an F1 score of 95.5% on an out-of-domain testset compared to a score of 75% for models trained on public datasets.
1. Introduction
The proliferation of open-domain conversational AI systems, such as Alexa and Google Assistant, has increased the frequency of interactions involving sensitive or offensive user inputs. Detecting such language is challenging because simple blacklists fail to capture implicit abuse, sarcasm, and context-dependent insults that bypass traditional keyword-based filters. Furthermore, existing supervised datasets for hate speech detection are often limited in scale and do not generalize well to the informal, spoken-language style of human-chatbot interactions. This work addresses these challenges by introducing a two-stage semi-supervised pipeline to generate a massive, high-quality training dataset from public web resources.
2. Methodology: Two-Stage Semi-Supervision
The core of our approach is a Two-Stage Semi-Supervised Pipeline designed to bootstrap a large-scale dataset from Reddit, leveraging the platform's community-based structure.
2.1 Stage 1: Blacklist-based Subreddit Ranking
We utilize a curated blacklist of over 800 sensitive terms to rank approximately 5,000 subreddits based on their overall sensitivity (the ratio of blacklist terms to total words). By sampling utterances from the most and least sensitive subreddits, we create an initial "Bootstrap" dataset to train a preliminary classifier.
2.2 Stage 2: Weakly Supervised Sampling
In the second stage, a binary classifier is trained on the public Kaggle "Toxic Comments" dataset. This model acts as a weak supervisor, working alongside the blacklist to filter the ranked subreddits. This process identifies 10 million high-confidence sensitive utterances and 10 million high-confidence non-sensitive utterances, resulting in a robust 20-million utterance dataset for final model training.
3. Experimental Setup and Datasets
We evaluate our models across several datasets to measure in-domain performance and out-of-domain generalization.
- Reddit: A 20 million utterance corpus used for the semi-supervised bootstrapping process.
- Toxic Comments (Wikipedia): Approximately 160k annotated sentences used for weak supervision and in-domain testing.
- Twitter: A dataset of ~25k annotated tweets used as a baseline.
- Chatbot Dataset: A manually annotated out-of-domain test set consisting of ~6,000 spoken utterances from a large-scale chatbot competition. This dataset includes multi-class labels such as Inappropriate, Insulting, and Racist.
4. Results and Discussion
Our "Two-Stage (TS) Bootstrap" model demonstrates significant improvements over baseline models trained on smaller, manually annotated datasets.
4.1 Out-of-Domain Generalization
On the Chatbot test set, the TS Bootstrap model achieved an F1 score of 95.5%. In contrast, models trained solely on Twitter or Toxic Comments datasets achieved F1 scores of 74.9% and 74.0%, respectively. This highlights the importance of scale and conversational context provided by the Reddit-bootstrapped data.
4.2 Addressing the Recall Gap
Traditional blacklists are precise but suffer from low recall (24.6% on the Chatbot dataset). The TS Bootstrap model effectively addresses this, achieving 92.0% recall. Furthermore, across seven distinct sensitive sub-classes (e.g., hate speech, pornographic content), the model maintained a weighted average recall of 95.5%, compared to 73.2% for baseline models.
5. Conclusion
This research demonstrates that leveraging community-level topical clustering through semi-supervision can overcome the limitations of manual annotation and static blacklists. By training on a massive, context-rich conversational dataset, we can build robust moderation systems that generalize across diverse sensitive categories and handle the nuances of open-domain human-chatbot interaction.
Summary updated