A deep dive into the consistently toxic 1% of Twitter

Hina Qayyum, Benjamin Zi Hao Zhao, Ian D. Wood, Muhammad Ikram, Mohamed Ali Kâafar, Nicolas KourtellisarXiv 2022 · OpenReview

TL;DR

This paper introduces the Twitter Toxic Tweets (3T) dataset, a massive collection of 293 million tweets from 122,000 users over 14 years, annotated for toxicity using Google's Perspective API. By analyzing the most consistently toxic users, the study reveals distinct patterns in their content, behavior, and potential for bot-like activity, offering a new lens for understanding influence operations on social media.

What Problem Does This Paper Address?

Online social networks (OSNs) are rife with misbehavior, from bullying to hate speech, which can be weaponized for influence operations. While much research focuses on identifying toxic content, there's a gap in understanding how consistently certain profiles engage in this behavior and what characteristics define these persistent offenders. This consistency is crucial for distinguishing genuine users from coordinated efforts aimed at manipulating public discourse. The authors aim to fill this gap by characterizing users who consistently post toxic content, moving beyond single-tweet classifications to a longitudinal analysis of user behavior.

Methods

The core of the methodology involves creating and analyzing the Twitter Toxic Tweets (3T) dataset. This dataset was built by:

  1. Curating Seed Datasets: Seven publicly available datasets with human annotations for toxic content were used as a starting point.
  2. User Timeline Crawl: Twitter's API was used to retrieve historical tweets (up to 3,200 per profile) for users identified in the seed datasets.
  3. Perspective API Annotation: Google's Perspective API was employed to assign toxicity scores (Toxicity, Severe Toxicity, Identity Attack, Inflammatory, Insult, Threat) to each of the 293 million tweets. This is a key differentiator, as it provides a granular, large-scale annotation absent in many prior studies.
  4. Focus Profile Identification: Users were identified as "focus profiles" if they exhibited consistently high toxicity (median Perspective score above a threshold) and low variability in their toxicity scores (high Gini index). This approach specifically targets persistent offenders.
  5. Content and Behavioral Analysis: The focus profiles were then analyzed for their shared URLs, domains, hashtags, tweet topics (using LDA), and posting patterns (timing, frequency).
  6. Botometer Analysis: To assess automation, profiles were queried using Botometer to estimate their likelihood of being bots.

Datasets

The primary dataset introduced is the Twitter Toxic Tweets (3T) dataset.

  • Size: 122,255 Twitter profiles, 293,401,161 individual tweets.
  • Time Span: 2007 to 2021 (14 years).
  • Annotation: Tweets are annotated with scores from six Google Perspective API models (Toxicity, Severe Toxicity, Identity Attack, Inflammatory, Insult, Threat).
  • Seed Datasets: Seven existing public datasets were used to bootstrap the user identification process.

Key Insights and Ideas

  • Consistency is Key: The paper highlights that focusing on consistent toxic behavior, rather than just isolated instances, is a powerful way to identify potentially malicious actors.
  • Toxic Echo Chambers: Consistently toxic profiles tend to operate within narrow thematic bubbles, sharing a limited diversity of URLs, domains, and hashtags, suggesting a coordinated or self-reinforcing environment.
  • Bot-like Behavior is Prevalent: The analysis strongly suggests that a significant portion of consistently toxic profiles exhibit characteristics indicative of automation, including regular posting times and high Botometer scores.
  • Evolution of Toxicity: The study reveals an increasing trend in toxic behavior on Twitter over the 15-year period, particularly in the years leading up to 2020.

Key Results

  • Dataset Scale: The 3T dataset is presented as the largest publicly available dataset annotated with Perspective API scores for online misbehavior.
  • Focus Profile Characteristics:
    • Consistently toxic profiles share a narrower range of domains but a larger set of URLs compared to random profiles.
  • They use fewer hashtags, and these hashtags are often malignant and toxic (e.g., related to war, conflict, or political extremism).
  • Tweet topics are highly cohesive and directly related to the type of misbehavior (e.g., Identity Attack profiles discussing the Israeli-Palestinian conflict).
  • Tweets exhibit lower comprehensibility and readability, with poorer vocabulary.
  • Posting behavior is highly regular and consistent, with small inter-tweet intervals and extended activity hours, often deviating from typical diurnal patterns.
  • Bot Likelihood: Focus profiles consistently score higher on Botometer, indicating a greater likelihood of being automated accounts. For instance, focus Identity Attack profiles had an average Botometer score of 4.53, with a 89.7% probability of being a bot, compared to random profiles with a score of 1.58 and a 57.8% bot probability.
  • Toxicity Trends: A linear regression model shows a statistically significant increase in all six measured toxicity dimensions over time, with positive slopes indicating rising misbehavior.

Limitations and Future Work

  • Obscene Content Removal: The authors acknowledge that initially, some toxic profiles were heavily involved in sharing obscene content. They manually removed these to focus on influence operations rather than general obscenity, which might exclude some relevant profiles.
  • API Limitations: The study relied on Google's Perspective API, which has its own biases and limitations. The authors note that the API's language support was also a constraint.
  • Future Work: The authors plan to further investigate the phenomenon of consistent, highly toxic misbehavior by analyzing profile self-declared locations and developing ML classifiers for automatic detection across platforms. They also aim to explore how these profiles are detected and removed by OSNs.

Summary updated

Abstract

Misbehavior in online social networks (OSN) is an ever-growing phenomenon. The research to date tends to focus on the deployment of machine learning to identify and classify types of misbehavior such as bullying, aggression, and racism to name a few. The main goal of identification is to curb natural and mechanical misconduct and make OSNs a safer place for social discourse. Going beyond past works, we perform a longitudinal study of a large selection of Twitter profiles, which enables us to characterize profiles in terms of how consistently they post highly toxic content. Our data spans 14 years of tweets from 122K Twitter profiles and more than 293M tweets. From this data, we selected the most extreme profiles in terms of consistency of toxic content and examined their tweet texts, and the domains, hashtags, and URLs they shared. We found that these selected profiles keep to a narrow theme with lower diversity in hashtags, URLs, and domains, they are thematically similar to each other (in a coordinated manner, if not through intent), and have a high likelihood of bot-like behavior (likely to have progenitors with intentions to influence). Our work contributes a substantial and longitudinal online misbehavior dataset to the research community and establishes the consistency of a profile's toxic behavior as a useful factor when exploring misbehavior as potential accessories to influence operations on OSNs.
BibTeX
@misc{qayyum2022deep,
  author = {Hina Qayyum and Benjamin Zi Hao Zhao and Ian D. Wood and Muhammad Ikram and Mohamed Ali Kâafar and Nicolas Kourtellis},
  title = {{A deep dive into the consistently toxic 1\% of Twitter}},
  year = {2022},
  eprint = {2202.07853},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.07853},
}

Loading related content...

Figure