BiRDy: Bullying Role Detection in Multi-Party Chats

Anaïs Ollagnier, Elena Cabrio, Serena Villata, Sara TonelliAAAI 2023 · DOI

Abstract

BiRDy is a novel web-based platform presented at AAAI-23 for detecting participant roles in cyberbullying episodes within private, multi-party chats. By leveraging fine-tuned multilingual BERT (mBERT) and a contextual window strategy, BiRDy identifies roles such as "bully," "victim," and "defender." The system addresses the unique challenges of private messaging, including data privacy and conversation entanglement, achieving high F-scores in both French and Italian datasets.

Introduction

While cyberbullying research often focuses on public platforms like Twitter or Reddit, a significant portion of teen aggression occurs in private instant messaging (IM) apps like WhatsApp or Messenger. These environments pose two primary challenges: the difficulty of collecting private data and the "semi-asynchronous" nature of chat logs, where multiple conversation threads overlap. BiRDy aims to overcome these hurdles by focusing on textual content and temporal structure to map participant dynamics.

Methods

The BiRDy system utilizes a two-stage pipeline for role detection:

  1. Message Labeling: Individual messages are classified using a fine-tuned mBERT model. To handle conversation entanglement, a contextual window strategy is employed, where the model considers the $n$ previous messages from the same user to capture the narrative arc.
  2. Role Assignment: A role scoring function aggregates message probabilities to assign each participant a single primary role (e.g., Bully, Victim, Defender, or Bystander Assistant).

The study utilized two datasets created through role-playing simulations: CyberAgressionAdo-v1 (French, 2,921 entries) and an Italian WhatsApp-style dataset (2,114 entries). These datasets include manual annotations for nuanced roles beyond simple binary classification.

Results and Discussion

Experimental results demonstrate that context is vital for accurate detection. Performance generally improves as the contextual window size increases, peaking at a window of 9 for French and 10 for Italian.

F-score performance across different contextual window sizes for French and Italian datasets.

The model achieved weighted F-scores of 0.778 for French and 0.869 for Italian. However, performance varied significantly by role and language. In the Italian dataset, bullies were identified with high accuracy (reaching an F-score of 1.0 in some settings), whereas victims were more easily identified in the French dataset.

Comparison of F-scores for Bully and Victim roles across French and Italian datasets as a function of contextual window size.

These variations suggest that the linguistic "scripts" of bullying and victimization may differ across cultures or specific training scenarios. Misclassifications typically occurred between linguistically similar roles, such as "bullies" and "bystander assistants."

Conclusion

BiRDy provides a robust framework for understanding the dynamics of private group chats. Future work will address the fluidity of roles over time, expand the system to more languages, and evaluate the model on larger, more diverse datasets.

Demo available at: http://134.59.134.227/demo_prd/index.html

Summary updated

BibTeX
@inproceedings{ollagnier2023birdy,
  author = {Anaïs Ollagnier and Elena Cabrio and Serena Villata and Sara Tonelli},
  title = {{BiRDy: Bullying Role Detection in Multi-Party Chats}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27080},
}
Figure