TL;DR
This paper introduces the Twitter Toxic Tweets (3T) dataset, a massive collection of 293 million tweets from 122,000 users over 14 years, annotated for toxicity using Google's Perspective API. By analyzing the most consistently toxic users, the study reveals distinct patterns in their content, behavior, and potential for bot-like activity, offering a new lens for understanding influence operations on social media.
What Problem Does This Paper Address?
Online social networks (OSNs) are rife with misbehavior, from bullying to hate speech, which can be weaponized for influence operations. While much research focuses on identifying toxic content, there's a gap in understanding how consistently certain profiles engage in this behavior and what characteristics define these persistent offenders. This consistency is crucial for distinguishing genuine users from coordinated efforts aimed at manipulating public discourse. The authors aim to fill this gap by characterizing users who consistently post toxic content, moving beyond single-tweet classifications to a longitudinal analysis of user behavior.
Methods
The core of the methodology involves creating and analyzing the Twitter Toxic Tweets (3T) dataset. This dataset was built by:
- Curating Seed Datasets: Seven publicly available datasets with human annotations for toxic content were used as a starting point.
- User Timeline Crawl: Twitter's API was used to retrieve historical tweets (up to 3,200 per profile) for users identified in the seed datasets.
- Perspective API Annotation: Google's Perspective API was employed to assign toxicity scores (Toxicity, Severe Toxicity, Identity Attack, Inflammatory, Insult, Threat) to each of the 293 million tweets. This is a key differentiator, as it provides a granular, large-scale annotation absent in many prior studies.
- Focus Profile Identification: Users were identified as "focus profiles" if they exhibited consistently high toxicity (median Perspective score above a threshold) and low variability in their toxicity scores (high Gini index). This approach specifically targets persistent offenders.
- Content and Behavioral Analysis: The focus profiles were then analyzed for their shared URLs, domains, hashtags, tweet topics (using LDA), and posting patterns (timing, frequency).
- Botometer Analysis: To assess automation, profiles were queried using Botometer to estimate their likelihood of being bots.
Datasets
The primary dataset introduced is the Twitter Toxic Tweets (3T) dataset.
- Size: 122,255 Twitter profiles, 293,401,161 individual tweets.
- Time Span: 2007 to 2021 (14 years).
- Annotation: Tweets are annotated with scores from six Google Perspective API models (Toxicity, Severe Toxicity, Identity Attack, Inflammatory, Insult, Threat).
- Seed Datasets: Seven existing public datasets were used to bootstrap the user identification process.
Key Insights and Ideas
- Consistency is Key: The paper highlights that focusing on consistent toxic behavior, rather than just isolated instances, is a powerful way to identify potentially malicious actors.
- Toxic Echo Chambers: Consistently toxic profiles tend to operate within narrow thematic bubbles, sharing a limited diversity of URLs, domains, and hashtags, suggesting a coordinated or self-reinforcing environment.
- Bot-like Behavior is Prevalent: The analysis strongly suggests that a significant portion of consistently toxic profiles exhibit characteristics indicative of automation, including regular posting times and high Botometer scores.
- Evolution of Toxicity: The study reveals an increasing trend in toxic behavior on Twitter over the 15-year period, particularly in the years leading up to 2020.
Key Results
- Dataset Scale: The 3T dataset is presented as the largest publicly available dataset annotated with Perspective API scores for online misbehavior.
- Focus Profile Characteristics:
- Consistently toxic profiles share a narrower range of domains but a larger set of URLs compared to random profiles.
- They use fewer hashtags, and these hashtags are often malignant and toxic (e.g., related to war, conflict, or political extremism).
- Tweet topics are highly cohesive and directly related to the type of misbehavior (e.g., Identity Attack profiles discussing the Israeli-Palestinian conflict).
- Tweets exhibit lower comprehensibility and readability, with poorer vocabulary.
- Posting behavior is highly regular and consistent, with small inter-tweet intervals and extended activity hours, often deviating from typical diurnal patterns.
- Bot Likelihood: Focus profiles consistently score higher on Botometer, indicating a greater likelihood of being automated accounts. For instance, focus Identity Attack profiles had an average Botometer score of 4.53, with a 89.7% probability of being a bot, compared to random profiles with a score of 1.58 and a 57.8% bot probability.
- Toxicity Trends: A linear regression model shows a statistically significant increase in all six measured toxicity dimensions over time, with positive slopes indicating rising misbehavior.
Limitations and Future Work
- Obscene Content Removal: The authors acknowledge that initially, some toxic profiles were heavily involved in sharing obscene content. They manually removed these to focus on influence operations rather than general obscenity, which might exclude some relevant profiles.
- API Limitations: The study relied on Google's Perspective API, which has its own biases and limitations. The authors note that the API's language support was also a constraint.
- Future Work: The authors plan to further investigate the phenomenon of consistent, highly toxic misbehavior by analyzing profile self-declared locations and developing ML classifiers for automatic detection across platforms. They also aim to explore how these profiles are detected and removed by OSNs.
Summary updated