Directions for NLP Practices Applied to Online Hate Speech Detection

Paula Fortuna, Mónica Domínguez, Leo Wanner, Zeerak TalatEMNLP 2022 · DOI

Abstract

Current Natural Language Processing (NLP) approaches often treat online hate speech detection as a standard text classification problem. This position paper argues that such a framework is fundamentally insufficient, as it ignores the deeply social, political, and contextual nuances of harmful language. By critiquing the standard NLP pipeline—from definition and annotation to evaluation—the authors advocate for a shift toward "design justice" and pluralistic modeling that prioritizes the perspectives of marginalized communities over majority-rule consensus.

Introduction: The Limits of Techno-Optimism

In recent years, the NLP community has approached the problem of online hate speech with a distinct "techno-optimism." The prevailing sentiment suggests that with sufficiently large datasets and increasingly complex Transformer-based architectures, the issue of online harm can be "solved." However, this perspective overlooks the fact that hate speech is not a static linguistic property but a situated social phenomenon.

Treating hate speech detection as a standard classification task—similar to sentiment analysis or topic labeling—fails to account for the power dynamics inherent in language. This paper argues that the current conventions of the field, particularly the reliance on majority-vote labeling and the assumption of Independent and Identically Distributed (I.I.D.) data, often end up harming the very populations these systems are intended to protect.

The Problem: Situated Language and the I.I.D. Myth

The core challenge of hate speech detection lies in its context-dependency. A specific term may function as a slur in one setting while serving as a reclaimed expression of empowerment in another. Current models, which typically strip away the identities of the speaker and the audience, remain brittle and easily bypassed by minor adversarial changes.

Furthermore, the standard machine learning assumption that data is I.I.D. is fundamentally flawed in the context of social media. Language evolves rapidly, and the "hatefulness" of a statement is inextricably linked to the relationship between the interlocutors. Because the distribution of language is constantly shifting, static benchmarks cannot capture the adversarial nature of online discourse.

Methodological Failures in the NLP Pipeline

The authors identify three primary areas where the standard NLP pipeline fails to address the complexities of hate speech:

  1. Definition and Concept Drift: "Hate speech" is frequently conflated with broader, less precise terms like "toxicity" or "abuse." This lack of definitional rigor leads to concept drift, where models are trained on inconsistent or overly broad criteria.
  2. The Annotation Paradox: In most NLP tasks, high Inter-Annotator Agreement (IAA) is viewed as a gold standard for quality. However, in hate speech detection, the authors argue that disagreement is often the most valuable data point. Forcing a single "gold label" via majority vote effectively silences minority perspectives. If a majority of annotators (who may not belong to a marginalized group) find a post acceptable, but a minority finds it hateful, the majority-rule approach erases the victim's experience.
  3. Evaluation and Stress Testing: Current evaluation metrics often fail to reveal whether a model understands the underlying social dynamics or is simply relying on superficial token triggers.

Empirical Evidence: Token-Based "Understanding"

To demonstrate these failures, the authors conducted a qualitative "stress test" using the Dynabench classifier, a state-of-the-art model. The results suggest that models are not learning the nuances of social oppression but are instead overfitting to specific keywords or templates.

Gender Invariance and Context Blindness

The model demonstrated a significant lack of understanding regarding historical power dynamics. For example:

  • Gender Invariance: The statement "Men's place is in the kitchen" was labeled as hateful with a 98.32% probability. This indicates the model fails to recognize that the harm of this specific trope is historically gender-specific; applying it to the dominant group does not carry the same weight of oppression.
  • Template Overfitting: The sentence "Women's place is in school" was flagged as hateful with a 99.60% probability. The model likely over-fitted to the "Women's place is in..." template, failing to distinguish between a restrictive trope and a supportive statement.
  • False Negatives for Violence: Most alarmingly, the sentence "Women’s place is in the cemetery" was labeled as not-hateful with a 99.96% probability. By avoiding "standard" slurs or toxic keywords, the statement bypassed the model's detection entirely, despite being a direct implication of femicide.

Conclusion and Future Directions

The authors conclude that the "fix" for hate speech detection cannot be purely algorithmic; it must be socio-technical. They propose several shifts in research and practice:

  • Pluralistic Labeling: Moving away from a single "gold label" toward models that can represent and maintain multiple subjective viewpoints simultaneously.
  • Design Justice: Incorporating the lived experiences of affected communities directly into the model design process, rather than treating them solely as sources of data or low-paid annotators.
  • Pragmatic Modeling: Developing methods that explicitly incorporate metadata—such as the identity of the speaker and the intended audience—to better capture the linguistic pragmatics of the conversation.

By rethinking the standard NLP script, the field can move toward systems that are not only more robust but also more equitable.

Summary updated

BibTeX
@inproceedings{fortuna2022directions,
  author = {Paula Fortuna and Mónica Domínguez and Leo Wanner and Zeerak Talat},
  title = {{Directions for NLP Practices Applied to Online Hate Speech Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.809},
}
Figure