Abstract
As conversational AI becomes a staple of modern social interaction, these systems are increasingly subjected to bullying and sexual harassment. This study investigates the response strategies of 11 state-of-the-art conversational agents—including commercial assistants like Alexa and Siri, rule-based systems, and data-driven models—to various forms of abuse. Our findings indicate a prevalence of nonsensical (40.5%) and flirtatious (22%) responses. Notably, data-driven models often produce inappropriate replies even when trained on "clean" datasets, suggesting that the issue of AI harassment is deeply rooted in contextual interpretation rather than just training data quality.
Introduction
Conversational AI systems are no longer just functional tools; they are perceived as social agents. This shift has led to a significant volume of user interactions involving sexual harassment. Addressing this behavior is crucial, as passive or flirtatious responses from AI—often personified with female names and voices—can reinforce harmful gender stereotypes and desensitize users to abusive behavior in real-world human relationships.
Methodology
The researchers systematically tested 11 AI systems across three categories:
- Commercial: Alexa, Siri, Google Home, and Cortana.
- Rule-based: Classic bots like ELIZA and ALICE.
- Data-driven: Models like Cleverbot and a custom sequence-to-sequence (Seq2Seq) model.
Harassment was categorized into four types: Gender and Sexuality, Sexualized Comments, Sexualized Insults, and Sexual Requests. To test the impact of data quality, the authors trained a "clean" Seq2Seq model on 20,000 Reddit utterance pairs filtered for profanity.
Results and Analysis
The study revealed a wide range of response strategies, many of which fail to adequately address harassment.
As shown in the figure above, the most frequent response category was "Nonsensical" (40.5% total), comprising non-coherent, ungrammatical, and "don't know" replies. Flirtation was the second most common response at 22%, primarily driven by adult-oriented bots but also appearing in standard data-driven and rule-based systems.
System-Specific Strategies
Response strategies varied significantly by system type:
- Commercial Systems: These agents typically favored "safe" strategies such as deflection, search results, or polite refusals. While avoiding direct conflict, this "playing innocent" persona can inadvertently reinforce submissive stereotypes.
- Data-driven Systems: These models were most prone to non-coherent replies, which users often misinterpret as being coy or flirtatious. They were also the most likely to retaliate with counter-aggression.
- Rule-based Systems: These systems frequently utilized intentional deflection to shift the topic away from harassment.
The categorical breakdown illustrates how these strategies shift depending on the nature of the harassment. For instance, "Sexualised Insults" often trigger a high rate of "No-answer" or "Non-coherent" responses from data-driven models, while "Sexualised Comments" see a spike in "Don't know" responses from commercial systems.
The "Clean Data" Fallacy
A key finding of this research is that "clean" training data is not a panacea. The model trained on filtered Reddit data still produced responses that could be interpreted as flirtatious (e.g., responding to "How big are your boobs?" with "How about you ask me?"). This suggests that the problem is not merely the presence of "dirty" words in training sets, but how humans project meaning onto ambiguous AI outputs.
Conclusion
Current conversational AI systems are ill-equipped to handle sexual harassment in a way that discourages abuse or avoids reinforcing harmful stereotypes. Future work must move beyond simple data filtering toward the development of ethical evaluation metrics and more sophisticated, assertive response strategies that prioritize social responsibility.
Summary updated