Challenges in Discriminating Profanity from Hate Speech

Shervin Malmasi, Marcos ZampieriarXiv 2018 · arXiv

TL;DR

Distinguishing targeted hate speech from general, non-antagonistic profanity on social media is a critical but frequently overlooked challenge in NLP. This paper evaluates a variety of surface features and advanced ensemble architectures on a three-class dataset, demonstrating that while meta-classification (stacked generalization) can push classification accuracy to nearly 80%, surface-level features fundamentally struggle with the semantic nuances and subjective nature of toxic language.

What Problem Does This Paper Address?

Most automated content moderation systems treat toxic language detection as a binary classification task: separating "hateful" or "bullying" content from "clean" content. However, this setup introduces a major flaw: it conflates general profanity with targeted hate speech.

Profanity is often used benignly in informal communication for emphasis or casual expression (e.g., "Holy shit, look at these prices"). In contrast, hate speech explicitly denigrates a protected individual or group. Blanket-filtering all profane language is both impractical and undesirable for open internet platforms. To build smarter moderation tools, we need systems that can accurately separate non-insulting profanity from targeted insults. This paper tackles this exact three-way classification problem: discriminating between Hate Speech (HATE), Offensive Language but no hate speech (OFFENSIVE), and Clean/Safe content (OK).

Methods

The authors systematically evaluate several feature spaces and model architectures using a linear Support Vector Machine (SVM) via the LIBLINEAR package as their core engine.

1. Feature Engineering

  • Surface n-grams: Character $n$-grams (lengths 2–8) and word $n$-grams (lengths 1–3).
  • Word Skip-grams: 1-, 2-, and 3-skip word bigrams to capture longer-distance dependencies.
  • Word Representations: Hierarchical bitstrings generated via Brown Clustering on a massive external corpus of unlabeled tweets to group syntactically and semantically similar words.

2. Architecture Variations

  • Single Classifiers: Models trained on individual feature types or a massive concatenated space of all 5.5 million features.
  • Parallel Classifier Ensembles: Combining 16 independent base classifiers (each trained on a distinct feature type) using fusion rules like Plurality Voting, Mean Probability, Median Probability, and Borda Count.
Figure 1: An example of parallel classifier ensemble architecture where N independent classifiers provide predictions which are then fused using an ensemble combination method. This can be considered a type of late fusion scheme. More specifically, in our experiments each classifier is a linear SVM trained on a single feature type.
  • Meta-Classification (Stacked Generalization): Instead of using fixed rules, the continuous probability outputs from all 16 base classifiers (48 features total) are fed into a second-level meta-classifier (evaluating both a Linear SVM and an RBF-kernel SVM) to learn how to optimally combine the base models' predictions.

Datasets

The study utilizes a dataset introduced by Davidson et al. (2017), consisting of 14,509 English tweets. Each tweet was reviewed by at least three human annotators to establish an adjudicated gold standard label.

The class distribution is heavily imbalanced, reflecting real-world social media distributions:

Dataset class distribution

For the unsupervised word representations, the authors leveraged a collection of 1,000 hierarchical clusters derived from 56 million English tweets (837 million tokens) by Owoputi et al. (2013).

Key Insights and Ideas

  • The Shared-Vocabulary Trap: Surface features are highly effective at separating "OK" text from toxic text, but they hit a wall when separating profanity from hate speech. Because both classes rely heavily on coarse vocabulary, simple bag-of-words models get confused.
  • Targeting is the Differentiator: Feature analysis revealed that hate speech is uniquely characterized by words targeting specific ethnic, religious, or social groups. Interestingly, the model picked up on subtle sociolinguistic nuances: the word "nigger(s)" was a top predictor for hate speech, whereas its variant "nigga(s)" was predominantly found in the general offensive category, indicating a difference in casual vs. derogatory usage.
  • Stacking Beats Simple Voting: Traditional ensemble methods (like voting or averaging) failed to outperform the best single classifier. However, meta-classification using an RBF-kernel SVM successfully mapped the systematic confusions of the base classifiers, yielding a significant performance boost.
  • The Subtlety of Hate: Classifiers completely fail when hate speech is expressed without profanity. For example, the tweet "Girls like you need to be 6 feet under" contains no colorful language or slurs, leading the models to misclassify it as "OK," despite its clear status as hate speech.

Key Results

  • Baseline & Upper Bound: The majority class baseline (always predicting "OK") yields 50.1% accuracy. An "Oracle" ensemble (which counts a prediction as correct if any single classifier gets it right) establishes a theoretical ceiling of 91.6%.
  • Best Single Model: Character 4-grams achieved the highest individual feature performance at 78.0% accuracy. Merging all features into one massive space did not improve this score (77.5%).
  • Meta-Classifier Triumph: The RBF-kernel SVM meta-classifier achieved the top overall accuracy of 79.8%, outperforming all voting ensembles and single models.
  • The Hardest Class: Error analysis via confusion matrices showed that the HATE class is by far the most difficult to pin down, yielding a poor recall of just 0.36 and an F1-score of 0.45. It is most frequently misclassified as general offensive language.

Limitations and Future Work

The authors highlight two major roadblocks for current systems:

  1. Surface Feature Limitations: Character and word $n$-grams cannot capture deep semantic meaning or intent. Future work must integrate deeper linguistic processing, such as dependency parsing and semantic parsing, to detect implicit threats and structural insults.
  2. Annotation Noise and Subjectivity: Toxic language is inherently subjective. A manual review revealed numerous instances where the machine's prediction was arguably more accurate than the human "gold" label. The authors recommend increasing the minimum number of annotators, providing stricter explicit guidelines, and filtering out non-native or low-proficiency speakers to improve dataset reliability.

Summary updated

Abstract

In this study we approach the problem of distinguishing general profanity from hate speech in social media, something which has not been widely considered. Using a new dataset annotated specifically for this task, we employ supervised classification along with a set of features that includes n-grams, skip-grams and clustering-based word representations. We apply approaches based on single classifiers as well as more advanced ensemble classifiers and stacked generalization, achieving the best result of 80% accuracy for this 3-class classification task. Analysis of the results reveals that discriminating hate speech and profanity is not a simple task, which may require features that capture a deeper understanding of the text not always possible with surface n-grams. The variability of gold labels in the annotated data, due to differences in the subjective adjudications of the annotators, is also an issue. Other directions for future work are discussed.
BibTeX
@misc{malmasi2018challenges,
  author = {Shervin Malmasi and Marcos Zampieri},
  title = {{Challenges in Discriminating Profanity from Hate Speech}},
  year = {2018},
  eprint = {1803.05495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.05495},
}

Loading related content...

Figure