Context Sensitivity Estimation in Toxicity Detection

Alexandros Xenos, John Pavlopoulos, Ion AndroutsopoulosACL 2021

TL;DR

This paper tackles the challenge of context-sensitive toxicity in online discussions, where a post's toxicity can hinge on its preceding message. They introduce a new task, "context sensitivity estimation," and a novel dataset (CCC) to address this, enabling the development of models that can identify posts whose toxicity is heavily influenced by their conversational context.

What Problem Does This Paper Address?

Online toxicity detection systems often treat each post in isolation, ignoring the conversational flow. This is problematic because many posts are only toxic because of what came before them. For example, "Get out!" might be aggressive on its own, but if the previous post was "This is the worst movie ever, I'm leaving," then "Get out!" could be a simple agreement. Current toxicity datasets are largely devoid of this crucial context, making it difficult for models to learn to identify such context-dependent toxicity. This paper aims to bridge this gap by focusing on how to detect and potentially leverage this context-sensitivity.

Methods

The core of the paper's methodology involves two main aspects:

  • Dataset Creation: They leverage the existing Civil Comments (CC) dataset and re-annotate 10,000 posts. Crucially, annotators were given two conditions: one where they only saw the target post (out-of-context, OC) and another where they also saw the preceding parent post (in-context, IC). This allows for a direct comparison of toxicity perception with and without context.
  • Context Sensitivity Estimation Task: They define a new task: predicting the difference in toxicity scores between the OC and IC annotations for a given post. This difference, denoted by δ, quantifies how much the perceived toxicity changes when context is considered. They then train various regression models (Linear Regression, SVR, Random Forest, and a BERT-based model) to predict this δ value.

Datasets

  • Civil Comments (CC) Dataset: The base dataset used for annotation.
  • Civil Comments in Context (CCC) Dataset: A newly constructed dataset of 10,000 posts, each with both in-context (IC) and out-of-context (OC) toxicity labels. This dataset is publicly released.

Key Insights and Ideas

  • Context-Sensitive Toxicity is Rare but Significant: While posts whose toxicity changes with context are not the majority (around 33.9% in their study), they represent a substantial portion. The degree of change, however, is often small.
  • Context Often Reduces Perceived Toxicity: A key finding is that providing the parent post often leads annotators to rate the target post as less toxic. This suggests that context can de-escalate or clarify potentially aggressive-sounding statements.
  • Standard Toxicity Detectors Struggle with Context-Sensitive Posts: When evaluated on subsets of posts with higher context sensitivity, standard toxicity detectors (like Perspective API) perform worse when context is not provided. This highlights the need for context-aware models.
  • Context Sensitivity Estimation is Feasible: The paper demonstrates that models can be trained to predict how sensitive a post's toxicity is to its context, with BERT-based models showing the best performance.

Key Results

  • Dataset Findings: In 66.1% of posts, toxicity scores remained unchanged. Of the remaining 33.9%, toxicity increased in 9.6% and decreased in 24.2% when context was provided.
  • Toxicity Detection Performance: When evaluating toxicity detection on subsets of posts with higher context sensitivity (t > 0.2), concatenating the parent post to the target post significantly reduced the Mean Absolute Error (MAE) for the Perspective API model. This benefit was not observed when considering all posts (t=0), as non-sensitive posts dominate.
  • Context Sensitivity Estimation Performance: The BERT-based regressor (BERTr) achieved the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) in predicting the context sensitivity score δ, outperforming other models. For classification of context-sensitive posts, BERTr also achieved the highest AUPR and AUC.

Limitations and Future Work

  • Limited Context Window: The study primarily focuses on the immediate parent post as context. Future work could explore longer conversational histories.
  • Focus on Toxicity Estimation, Not Detection: While they show context-sensitive posts are harder to detect with standard models, the primary task developed is estimating sensitivity, not directly improving toxicity detection using this estimation.
  • Future Work: The authors suggest using context sensitivity estimators to curate larger datasets of context-sensitive posts for training better toxicity detectors. They also propose investigating better context-aware models and exploring whether manually annotating context sensitivity itself can be done without the parent post.

Summary updated

BibTeX
@inproceedings{xenos2021context,
  author = {Alexandros Xenos and John Pavlopoulos and Ion Androutsopoulos},
  title = {{Context Sensitivity Estimation in Toxicity Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure