Detecting harassment in real-time as conversations develop

Wessel Stoop, Florian Kunneman, Antal van den Bosch, Ben MillerACL 2019

TL;DR

This paper introduces a novel machine learning approach to detect toxic behavior in online gaming conversations as they unfold, rather than just analyzing individual messages. Their method adapts a confidence threshold dynamically, starting low and increasing as the conversation progresses, to optimize the detection of harassment in real-time. This has significant implications for game developers looking to foster healthier online communities by enabling timely interventions.

What Problem Does This Paper Address?

Online gaming communities are plagued by verbal harassment, often termed "toxicity." This behavior not only degrades the player experience but also drives players away from games. While existing methods often focus on classifying individual chat messages, this approach is insufficient for real-time intervention in fast-paced games. Harassment in games is often subtle, develops over time, and relies on context that spans multiple turns. Game developers need a way to identify toxic players during a match to issue warnings, mute, or ban them, thereby preventing further escalation and improving community health. The challenge lies in detecting this evolving toxicity accurately and early enough to be actionable.

Methods

The authors developed a framework called HaRe (Harassment Recognizer) that treats conversation analysis as a dynamic, time-series problem. Instead of classifying each utterance in isolation, HaRe concatenates all utterances from a given player up to a certain point in the conversation and feeds this growing text into a classifier. They employ a sophisticated neural network architecture, specifically a Recurrent Neural Network (RNN) with an embedding layer, bidirectional GRU layers, and dense layers, similar to top performers in the Toxic Comment Classification Challenge.

The core innovation lies in their approach to the confidence threshold. They observe that a static threshold for classifying a player as toxic is suboptimal. Instead, they propose a sliding threshold: it should start low in the early stages of a conversation to catch nascent toxicity, and then gradually increase as more conversational data becomes available. This dynamic adjustment aims to balance precision (avoiding false positives) and recall (catching actual toxic players) over the course of a match. The rate at which this threshold should increase is shown to be influenced by the size of the training dataset.

Datasets

The study utilizes a dataset of 5000 conversations from the popular video game League of Legends, provided by Riot Games. This dataset includes utterances from 48,512 players. Toxic players were initially identified by teammates and opponents and then reassessed by the community through a voting system called the "Tribunal." Only cases with an "overwhelming majority" of votes were labeled as toxic. The dataset features an average conversation length of 186.77 utterances, with an average utterance length of 3.15 words. Notably, only 10.3% of speakers in this dataset were labeled toxic by the Tribunal, reflecting a class imbalance.

Key Insights and Ideas

  • Dynamic Thresholding is Crucial: The most significant finding is that a fixed confidence threshold for detecting toxicity is inefficient. An optimal strategy involves a "sliding threshold" that starts low and increases as the conversation progresses. This acknowledges that early interactions might be less indicative of persistent toxicity, while later interactions, with more context, can be more reliably assessed.
  • Training Data Size Matters: The rate at which the sliding threshold should increase is not constant. It interacts with the size of the training dataset. Larger training sets allow for a slower increase in the threshold, suggesting that the model becomes more robust and less prone to misinterpreting early utterances as toxic.
  • Context is King: The paper emphasizes that individual utterances are often insufficient for detecting nuanced harassment. The model's strength lies in its ability to process the cumulative context of a player's contributions over time.

Key Results

The paper presents visualizations (Figures 3, 4, and 5) showing precision, recall, and F-score over the course of conversations for various confidence thresholds. They demonstrate that recall generally increases as more utterances become available, while precision can decrease due to the increased possibility of misinterpretation. The F-score, a balance of precision and recall, is optimized by the proposed sliding threshold strategy. Figure 6 illustrates how the optimal threshold increases over time, with the rate of increase being slower for larger training sets (e.g., 4000 conversations vs. 1000).

Limitations and Future Work

The authors acknowledge a limitation regarding low recall scores in the early stages of conversations. These scores can be ambiguous, potentially indicating either a poorly performing classifier or a genuine lack of toxicity. They suggest that metrics like Area Under the ROC Curve (AUC) could provide a more comprehensive evaluation by assessing correct negative judgments.

Future work could involve comparing their approach to utterance-level classification methods to pinpoint the exact moment toxicity begins. Additionally, exploring other metrics that can better differentiate between a lack of toxicity and missed toxic instances would be beneficial. The paper also notes that the optimal parameters for the sliding threshold are dataset and classifier dependent, suggesting the need for further tuning and graph-based decision-making for practical deployment.

Summary updated

BibTeX
@inproceedings{stoop2019detecting,
  author = {Wessel Stoop and Florian Kunneman and Antal van den Bosch and Ben Miller},
  title = {{Detecting harassment in real-time as conversations develop}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure