Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop

Jonathan P. Chang, Cristian Danescu-Niculescu-MizilEMNLP 2019 · DOI

TL;DR

This paper introduces CRAFT, a novel model that forecasts the derailment of online conversations before they become toxic. By learning conversational dynamics in an unsupervised way and then fine-tuning for prediction, CRAFT significantly outperforms existing methods, offering valuable early warnings for human moderators.

What Problem Does This Paper Address?

Online communities are plagued by antisocial behavior, which can range from harassment to personal attacks. While many systems can detect toxic content after it appears, this is often too late for effective moderation. The challenge is to predict when a conversation is heading towards derailment, giving moderators a heads-up to intervene proactively. Existing approaches struggle with this because conversations are dynamic, their outcomes depend on the interplay of comments, and they have an unpredictable "horizon" – they can derail at any moment. This paper aims to bridge this gap by developing a model that can forecast derailment in an online fashion, as the conversation unfolds.

Methods

The core of the proposed solution is the CRAFT (Conversational Recurrent Architecture for ForecasTing) model, which integrates two key components:

  • Generative Component (Unsupervised Pre-training): This part uses a hierarchical recurrent encoder-decoder (HRED) architecture, similar to those used in chatbot development. The HRED model is trained on a large corpus of unlabeled conversations to learn the general dynamics and structure of online discussions. It encodes individual comments and then uses a context encoder to capture the evolving state of the conversation. This unsupervised pre-training allows the model to learn rich representations of conversational context without needing explicit labels for derailment.
  • Prediction Component (Supervised Fine-tuning): Once the generative component is pre-trained, its learned representations are used as a starting point for a supervised prediction task. A multi-layer perceptron (MLP) is added to predict the probability of derailment. This component is then fine-tuned on smaller, labeled datasets where conversations are marked as either derailing or not. This "pre-train then fine-tune" paradigm is crucial for tackling the scarcity of labeled data in this domain.

A key innovation is CRAFT's ability to process conversations "online," meaning it updates its prediction with each new comment, addressing the unknown horizon problem.

Datasets

The authors introduce and utilize two datasets for their research:

  • Wikipedia Conversations ("Conversations Gone Awry" dataset): This is an expanded version of an existing dataset. It consists of Wikipedia talk page conversations that were crowd-labeled to ensure they were civil before a personal attack occurred. The authors expanded this dataset to include longer conversations (4+ comments before derailment) and released it as part of the Cornell Conversational Analysis Toolkit (ConvoKit).
  • Reddit CMV (ChangeMyView) dataset: This is a newly constructed dataset from the subreddit ChangeMyView. Unlike the Wikipedia dataset, it uses moderator actions (comments removed for violating rules against rudeness or hostility) as a proxy for derailment, avoiding post-hoc annotations. This dataset aims to capture more "in-the-wild" derailment scenarios.

Both datasets were split into training, development, and testing sets, with paired conversations kept together to control for topic. A large corpus of unlabeled conversations was also used for unsupervised pre-training.

Key Insights and Ideas

  • Derailment as an Emerging Property: The paper frames derailment not as an isolated toxic comment, but as an emergent property of the conversation's dynamics. This shift in perspective is fundamental to developing predictive models.
  • Pre-training for Contextual Understanding: The success of the model hinges on unsupervised pre-training of the generative component. This allows CRAFT to learn nuanced conversational patterns from vast amounts of unlabeled data, which is essential given the limited availability of labeled derailment data.
  • Online Prediction is Crucial: The ability to make predictions as the conversation unfolds is a critical requirement for practical moderation tools. CRAFT's architecture naturally supports this, updating its forecast with each new comment.
  • Order Matters: Experiments suggest that CRAFT doesn't just treat comments as a bag of words; it learns to leverage the order of comments to understand conversational dynamics, which is vital for detecting subtle shifts towards toxicity.

Key Results

  • State-of-the-Art Performance: CRAFT significantly outperforms existing fixed-window and online baselines on both the Wikipedia and Reddit CMV datasets, achieving higher accuracy and F1 scores.
  • Early Warning System: The model provides substantial early warning, triggering on average 3 comments (or 3 hours) before an overtly toxic comment appears on Wikipedia. For CMV, this is 4 comments. Over 39% of conversations receive at least 12 hours of early warning.
  • Impact of Pre-training: Training the prediction component without pre-training results in performance at the level of random guessing, highlighting the necessity of unsupervised pre-training.
  • Context Encoder Importance: A simplified version of CRAFT without the context encoder (effectively ignoring inter-comment dependencies) performs significantly worse, underscoring the value of capturing conversational dynamics.

Limitations and Future Work

  • Interpretability: The authors acknowledge that neural network models are often black boxes, and future work could focus on developing more transparent models to understand how CRAFT extracts order-related features.
  • Moderator Needs: The ideal "early warning" threshold and the optimal balance between precision and recall for moderators are open questions that could be addressed through user studies.
  • Dataset Balance: Derailment is a rare event, and the current analysis uses balanced datasets. Future work could explore how the model performs with more realistic, imbalanced data and different trigger thresholds.
  • Finer-Grained Forecasting: The current model assigns a single label (derailment or not) to an entire conversation. Future research could explore predicting more nuanced conversational trajectories, accounting for the possibility of recovery or repeated antisocial behavior.

Summary updated

BibTeX
@inproceedings{chang2019trouble,
  author = {Jonathan P. Chang and Cristian Danescu-Niculescu-Mizil},
  title = {{Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1481},
}
Figure