Introduction
Abusive language detection has traditionally focused on social media platforms like Twitter and Wikipedia. However, as Conversational AI systems become more integrated into daily life, they face a unique set of challenges. Most current systems rely on simplistic keyword-spotting, which often fails to distinguish between nuanced abuse and benign expressions of surprise or frustration. Furthermore, the anthropomorphization and gendering of AI agents can reinforce harmful stereotypes if abuse is not properly mitigated.
To address these gaps, researchers have introduced ConvAbuse, the first large-scale English dataset of "in the wild" abusive language directed at conversational AI. This dataset moves beyond binary labels to provide a nuanced, expert-annotated framework for understanding the specific dynamics of human-AI interaction.
Methodology
The authors developed a hierarchical annotation scheme designed to capture the complexity of conversational abuse. This scheme includes:
- Severity Scale: A Likert scale ranging from +1 (friendly) to -3 (strongly abusive).
- Target & Directness: Identification of whether abuse is explicit or implicit, and its intended target (the system, a group, or a third party).
- Abuse Type: Eight distinct categories, including sexism, sexual harassment, ableism, and intellectual attacks.
To ensure high-quality data, the study employed eight Gender Studies students as expert annotators. These experts were trained to recognize subtle forms of aggression and systemic bias that typical crowdsourced workers might overlook.
Dataset Analysis and Distribution of Abuse
The ConvAbuse corpus contains over 20,000 ratings across 6,837 unique examples from three systems: Alana v2 (a female-sounding social bot), CarbonBot (a task-oriented assistant), and ELIZA (a rule-based chatbot).
Analysis reveals a significant "female penalty." Systems with female names or voices (Alana and ELIZA) received substantially more sexual harassment and sexist abuse compared to the ungendered CarbonBot. As shown in Figure 1, over half of the abusive instances in the dataset involve sexism or sexual harassment, a stark contrast to social media datasets where abuse is often political or group-targeted.
Contextual information is also critical; many abusive turns are only identifiable when considering the bot's previous prompt. Users frequently employ "polite" phrasing for implicit attacks (e.g., "Are you stupid?"), which are harder for automated systems to detect than explicit profanity.
Benchmarking and Model Performance
The researchers evaluated several models, including SVM, MLP, and BERT, on the binary detection task. The results highlight a major "domain shift" problem. Models trained on Twitter data (OLID) performed poorly on ConvAbuse, with F1 scores dropping significantly.
As illustrated in Figure 6, fine-tuned BERT models achieved the highest performance, reaching macro F1 scores of up to 88.92%. However, the study also found that while training on massive datasets like Jigsaw (Wikipedia) can yield high scores due to sheer volume, models trained on domain-specific data (ConvAbuse) are far superior when training sizes are equalized. Despite these successes, identifying the specific type or severity of abuse remains a challenging task for current state-of-the-art models.
Conclusion and Future Work
ConvAbuse provides a vital resource for developing more sophisticated and socially responsible conversational agents. Future work will focus on improving multi-label classification to better distinguish between overlapping categories like sexism and sexual harassment. Additionally, the researchers aim to develop mitigation strategies that allow bots to respond to abuse without reinforcing harmful gender stereotypes.
Summary updated