Towards Automated Sexual Violence Report Tracking

Naeemul Hassan, Amrit Poudel, Jason G. Hale, Claire Hubacek, Khandaker Tasnim Huq, Shubhra Kanti Karmaker Santu, Syed Ishtiaque AhmedICWSM 2019

TL;DR

This paper introduces an automated system for tracking sexual violence reports from social media, aiming to overcome the challenges of manual detection and the inherent silence surrounding such incidents. By leveraging supervised learning and natural language processing, the model can identify, characterize, and even tag perpetrators in sexual violence reports with impressive accuracy, offering a scalable solution for understanding and addressing this critical issue.

What Problem Does This Paper Address?

Sexual violence is a pervasive global issue with severe consequences for victims. However, detecting and understanding its prevalence, nature, and patterns is incredibly challenging. This difficulty stems from several factors: the subjective and context-dependent nature of sexual violence, the reluctance of victims to report due to social stigma, cultural taboos, fear of retaliation, and victim-blaming. Traditional reporting mechanisms often fail to capture the full scope of the problem. While social media has emerged as a platform for victims to share their experiences, manually sifting through the vast amount of data to identify and analyze these reports is an insurmountable task. This paper addresses the need for an automated, scalable, and reliable system to track sexual violence reports from social media, thereby shedding light on this "silence" and providing valuable insights for support, policy, and prevention efforts.

Methods

The authors developed a two-step approach for automated sexual violence report tracking:

  1. Sexual Violence Report Detection: This is framed as a binary classification problem to distinguish between sexual violence reports (SVR) and non-sexual violence reports (nSVR). They employed a Majority Voting Classifier (MVC), which combines the predictions of Support Vector Machines (SVM), Random Forest (RDF), and Gradient Boosting (GDB) models. This ensemble approach aims to leverage the strengths of each individual classifier.
  2. Sexual Violence Report Characterization: Once a report is identified as SVR, the system characterizes it across several dimensions:
  3. Violence Category: Multi-class classification to categorize the type of sexual violence (e.g., penetration, unwanted sexual contact).
  4. Victim Identity: Binary classification to determine if the victim is self-reporting (SLF) or if the report is about another victim (nSLF).
  5. Victim Gender: Multi-class classification for victim gender (Female, Male, Unspecified).
  6. Perpetrator Category: Multi-class classification to identify the relationship between the victim and perpetrator (e.g., intimate partner, stranger).
  7. Perpetrator Tagging: This is treated as a sequence labeling problem, using a Conditional Random Field (CRF) model to identify and extract words from the tweet that describe the perpetrator.

A key preprocessing step involved using Semantic Role Labeling (SRL) to identify potential sexual violence-related verb-agent-detail tuples, which helped filter the dataset for relevant tweets before annotation.

Datasets

The primary dataset used in this study is a collection of approximately one million #MeToo hashtagged tweets gathered from October 15-31, 2017.

  • Publicly Available Subset: A 25% random sample of the #MeToo dataset was made publicly available for other researchers.
  • Annotated Data: Approximately 18,000 tweets were manually annotated using Amazon Mechanical Turk (AMT) following CDC guidelines. This annotated subset forms the benchmark for training and evaluating their models.
  • Post-MeToo Corpus: For comparative analysis, a separate corpus of 631,677 tweets was collected in May 2019, representing the period after the peak of the #MeToo movement, to assess reporting trends without an active online movement.

Key Insights and Ideas

  • Social Media as a Rich Source: The #MeToo movement demonstrated the power of social media for collective expression of experiences with sexual violence, providing a vast, albeit noisy, dataset for analysis.
  • Automated Tracking is Feasible and Necessary: The success of their automated system highlights the potential to overcome the limitations of manual analysis and the "silence" surrounding sexual violence.
  • Context is Crucial: Identifying sexual violence requires more than just keywords; understanding the victim's context, the perpetrator's relationship, and the nature of the act is vital.
  • Perpetrator-Violence Relationships: The analysis revealed interesting patterns, such as strangers being the most common perpetrators of unwanted sexual contact, while closer relationships (family, intimate partners) are more associated with severe forms of violence like penetration.
  • Self-Reporting Dominance: A significant portion of identified sexual violence reports were self-reported, underscoring the importance of social media as a reporting channel for victims.

Key Results

  • Sexual Violence Report Detection: The MVC classifier achieved a precision of 80.4% and a recall of 83.4%, with an F1-measure of 80.8% for detecting sexual violence reports.
  • Perpetrator Tagging: The CRF model achieved a weighted-average F1-measure of 76% for annotating perpetrator identity from reports.
  • Characterization Performance: The MVC achieved the highest weighted-average F1-measures across various characterization tasks: 0.57 for violence, 0.82 for victim identity, 0.612 for gender, and 0.55 for perpetrator category.
  • #MeToo Movement Analysis: The system identified over 106,000 tweets containing sexual violence reports from the #MeToo corpus, with a significant number being self-reported.
  • Post-MeToo Period Analysis: Even without an active movement, the system detected a substantial number of self-reported sexual violence tweets, indicating continued use of social media for reporting.

Limitations and Future Work

  • Feature Limitations: The current models primarily use unigram and bigram features. Future work could explore user-specific and network-specific features to improve characterization, especially for gender and victim identity.
  • Class Imbalance: Some characterization classes suffered from class imbalance, which impacted model performance. While undersampling and oversampling were explored, more data for these underrepresented classes is needed.
  • Credibility Assessment: The current system does not consider the credibility of a report, which could lead to false positives.
  • Adversarial Attacks: The authors acknowledge the potential for adversarial attacks and suggest a human-in-the-loop system to mitigate this.
  • Privacy Concerns: They highlight the need to support victims without necessarily contacting them through conventional channels, suggesting the creation of peer support communities.
  • Data Sharing: The authors plan to make a portion of their collected #MeToo data and annotated data publicly available, along with their code, to foster reproducibility and further research.

Summary updated

BibTeX
@inproceedings{hassan2019towards,
  author = {Naeemul Hassan and Amrit Poudel and Jason G. Hale and Claire Hubacek and Khandaker Tasnim Huq and Shubhra Kanti Karmaker Santu and Syed Ishtiaque Ahmed},
  title = {{Towards Automated Sexual Violence Report Tracking}},
  booktitle = {ICWSM},
  year = {2019},
}
Figure