ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI

Amanda Cercas Curry, Gavin Abercrombie, Verena Rieser

Researchers have introduced ConvAbuse, the first large-scale English corpus of abusive language directed at conversational AI "in the wild." The study reveals that abuse towards chatbots is fundamentally different from social media toxicity—it is more frequently sexist, sexually aggressive, and implicit. Crucially, the research demonstrates that current state-of-the-art models trained on social media data struggle to detect these nuances.

The Challenge of Conversational Abuse

Most research into abusive language detection focuses on social media platforms like Twitter or Wikipedia. However, humans interact with AI agents differently than they do with other humans. Because many AI assistants are anthropomorphized and often given female personas, they become targets for specific types of gender-based abuse.

Current industry solutions often rely on simple keyword spotting, which fails to catch subtle, "veiled" toxicity (false negatives) while over-flagging harmless frustration or slang (false positives). There is a critical lack of high-quality, "nuanced" data to train models that can handle the private, one-on-one nature of human-computer interaction.

Methodology and Annotation

The authors developed a hierarchical annotation scheme to capture the complexity of abuse. Instead of a simple "toxic/not toxic" binary, they used:

  • Severity Scale: A Likert scale from -3 (strongly abusive) to +1 (friendly).
  • Fine-grained Types: Labels for sexism, sexual harassment, intelligence-based attacks, racism, etc.
  • Target & Directness: Identifying who the abuse is aimed at and whether it is explicit or implicit.

To ensure high-quality labels, they recruited expert annotators (Gender Studies students) rather than general crowdsourced workers. They then benchmarked three types of models—Support Vector Machines (SVM), Multi-Layer Perceptrons (MLP), and BERT—to evaluate performance on this new data.

Dataset Characteristics

The core contribution is the ConvAbuse dataset, containing over 20,000 ratings on 6,837 unique samples. The data comes from three distinct types of "real-world" systems: Alana v2 (an open-domain social bot), CarbonBot (a task-oriented chatbot), and ELIZA (a rule-based psychotherapist bot).

Distribution of abuse types across different conversational agents, highlighting the prevalence of sexism and sexual harassment.

Key Insights

  • The "Persona" Matters: Bots with female names or voices (Alana v2 and ELIZA) received significantly more sexual harassment and sexist abuse than the gender-neutral CarbonBot.
  • Domain Shift: Models trained on Twitter or Wikipedia data perform poorly on chatbot data. Social media abuse is often directed at third parties, whereas chatbot abuse is 92% directed at the bot itself.
  • Implicit Abuse: In ConvAbuse, abuse is frequently subtle or phrased as a question (e.g., "can i be your lover?"). Detection is harder because the proportions of explicit vs. implicit abuse are more balanced than in social media datasets.
  • Contextual Necessity: Including the previous turns of the conversation helps models, as user intent often only becomes clear when viewed alongside the bot's prior response.

Experimental Results

Benchmarking results show that fine-tuned BERT models achieved the highest performance, with a macro F1 score of 88.92% on the binary detection task. However, when models trained on social media datasets (like OLID or Jigsaw) were tested on ConvAbuse, their performance dropped significantly, confirming the unique nature of conversational abuse.

Performance (F1 score) of various models across different training datasets, showing BERT as the leader and the poor transferability of social media models.

While models are decent at binary "abuse vs. no abuse" detection, they struggle with fine-grained tasks, frequently confusing "sexism" with "sexual harassment." Expert annotators reached high agreement on categories like racism ($\alpha=0.96$) but found "directness" ($\alpha=0.26$) and "sexism" ($\alpha=0.63$) much more subjective.

Limitations and Future Work

The authors acknowledge limitations regarding data release constraints due to privacy and the homogeneity of the annotator pool (primarily young, female-identifying students). Furthermore, rare types of abuse like transphobia or ableism remain difficult to detect due to small sample sizes in the "wild" data. Future work suggests moving toward multi-task learning frameworks that can model individual annotator perspectives rather than forcing a single "gold standard" label.

Summary updated

BibTeX
@inproceedings{curry2021convabuse,
  author = {Amanda Cercas Curry and Gavin Abercrombie and Verena Rieser},
  title = {{ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
}
Figure