Preemptive Toxic Language Detection in Wikipedia Comments Using Thread-Level Context

Vanja Mladen Karan, Jan ŠnajderACL 2019

TL;DR

This paper tackles the challenge of predicting toxic comments before they are posted by analyzing entire Wikipedia conversation threads. They introduce a new, large-scale dataset of Wikipedia conversations and explore how incorporating thread context impacts preemptive toxicity detection, finding that while context can help in specific scenarios, current models still struggle with practical application.

What Problem Does This Paper Address?

The internet is awash with user-generated content, making manual moderation of toxic language an overwhelming task. While automated systems can help, they typically operate after a toxic comment has already been made (post-hoc detection). This paper explores a more proactive approach: preemptive moderation. The goal is to identify conversation threads that are likely to lead to toxic comments in the future, allowing moderators to intervene early. A key challenge is understanding whether analyzing the entire conversation thread, rather than just individual comments, provides a significant advantage for this preemptive task.

Methods

The authors investigate two main extensions to standard toxic language detection:

  • Preemptive Classification: Instead of classifying a comment as toxic after it's posted, the model aims to predict if a future comment in a thread will be toxic.
  • Thread-Level Context: Traditional models often treat each comment in isolation. This work explores incorporating information from preceding comments within the same conversation thread.

For their models, they utilize deep learning approaches, specifically focusing on BiLSTM (Bidirectional Long Short-Term Memory) networks. They experiment with two model variants: one that is thread-agnostic (only considers the current comment) and one that is thread-aware (incorporates preceding comments). They also compare these to a simpler linear Support Vector Machine (SVM) model using TF-IDF features.

Datasets

The primary dataset used is a newly compiled, large-scale dataset of Wikipedia conversations. This dataset was derived from existing data (Hua et al., 2018) and enriched with:

  • Complete conversation threads: Including the entire history of comments.
  • Semi-automatically generated toxicity labels: Comments were flagged as toxic if a toxicity classifier indicated high confidence and the comment was deleted by someone other than the author. This heuristic aims to capture toxic comments that are likely to be removed.

They create several variants of this dataset, distinguishing between:

  • Real-context: Threads with at least two participants.
  • Rich-context: Threads filtered to ensure a minimum number of preceding comments (Lmin = 2) are available for context.
  • Post-hoc scenario: All comments in a thread are considered for classification.
  • Preemptive scenario: Comments are considered for predicting future toxicity.

Key Insights and Ideas

  • Preemptive detection is challenging but potentially valuable: While current models struggle to achieve high performance in the preemptive scenario, they can still beat random baselines and offer a glimpse into proactive moderation.
  • Thread context isn't always a silver bullet: For post-hoc detection, thread context offers less benefit than expected, as the toxicity is often evident within the comment itself. For preemptive detection, context becomes more important when a toxic comment has already occurred in the thread, but its utility diminishes when trying to predict toxicity from a completely clean slate.
  • The "silver labels" have limitations: The semi-automatic labeling process, while efficient, might miss toxic comments that are not deleted or are deleted for reasons other than toxicity. This could impact the model's ability to generalize to all types of toxic language.
  • Class imbalance is a major hurdle: The dataset is heavily skewed towards non-toxic comments, which significantly degrades model performance in practical settings.

Key Results

  • Post-hoc detection significantly outperforms preemptive detection: As expected, models perform much better when classifying existing toxic comments compared to predicting future ones.
  • BiLSTM-context models show marginal gains: In most settings, the thread-aware BiLSTM models did not significantly outperform their thread-agnostic counterparts. A notable exception was the preemptive "real-context" scenario, where context did provide a slight advantage.
  • SVMs can be competitive: Surprisingly, in some scenarios, a simpler SVM model with TF-IDF features outperformed the deep learning models, suggesting that complex thread context might not always be necessary or effectively utilized by the current deep learning architectures.
  • F1-scores are low for preemptive models: Even the best preemptive models achieved F1-scores that are too low for practical application, highlighting the difficulty of the task.

Limitations and Future Work

The authors acknowledge several limitations:

  • Silver label quality: The semi-automatic labeling might not capture all nuances of toxicity, and manual re-labeling with retuned thresholds could improve quality.
  • Class imbalance: The extreme imbalance in the dataset significantly impacts practical performance, and further work is needed to address this.
  • Limited thread context modeling: The current models use simple mechanisms for incorporating thread context. Future work could explore more sophisticated graph-based deep learning methods (e.g., GraphSAGE) to better model thread structures.
  • User-level information: Incorporating user-specific features (e.g., past toxic behavior, personality profiles) could enhance detection.
  • Discourse and pragmatic features: Combining deep learning with linguistic features related to discourse and pragmatics might improve preemptive detection.

Summary updated

BibTeX
@inproceedings{karan2019preemptive,
  author = {Vanja Mladen Karan and Jan Šnajder},
  title = {{Preemptive Toxic Language Detection in Wikipedia Comments Using Thread-Level Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure