Measuring and Characterizing Hate Speech on News Websites

Savvas Zannettou, Mai ElSherief, Elizabeth Belding, Shirin Nilizadeh, Gianluca StringhiniarXiv 2020 · arXiv

TL;DR

This paper presents a massive-scale study analyzing 125 million comments across 412,000 news articles to understand what triggers online toxicity. The researchers found that spikes in hate speech are heavily synchronized with divisive real-world political events and are often amplified by external "raids" from platforms like 4chan and Reddit. Furthermore, the linguistic style of a news article itself can reliably predict whether it will become a magnet for hateful discourse.

What Problem Does This Paper Address?

While news websites have become highly social spaces by allowing readers to comment and debate, these sections frequently devolve into toxic environments filled with racism and hate speech. Managing this is a massive headache for publishers, yet the research community has lacked a macro-level understanding of the dynamics at play.

The authors tackle several open questions: What kind of news content attracts hate speech? How do real-world political events shift the baseline of online toxicity? And how do external, highly polarized web communities (like 4chan's /pol/ or specific subreddits) influence the comment sections of mainstream media outlets?

Methods

The authors built a multi-stage pipeline to collect and analyze their massive dataset:

  1. Comment Scraping: They deployed custom web crawlers to extract comments from news sites utilizing various commenting architectures (Disqus, Spot.IM, Facebook, and custom setups).
  2. Partisanship Classification: News outlets were categorized into political leanings (Left to Extreme-Right) using annotations from Media Bias/Fact Check (MBFC).
  3. Hate Speech Detection: They evaluated multiple classifiers and selected Google’s Perspective API (specifically the Severe Toxicity model) due to its high precision and robustness against simple swear-word false positives.
  4. Temporal Analysis: To map toxicity spikes to real-world events, they used the Pruned Exact Linear Time (PELT) changepoint analysis algorithm to detect statistically significant shifts in time-series data.
  5. Linguistic Profiling: They leveraged the Linguistic Inquiry and Word Count (LIWC) tool to analyze the psychological and structural attributes of the news articles themselves.

Datasets

The study analyzes a newly compiled dataset spanning a 19-month period (July 2016 to February 2018):

  • 125 Million comments posted on 412,000 news articles across 64 popular news domains.
  • To ensure the articles were relevant to cross-platform discussions, the dataset was seeded by tracking all news links shared on 4chan's /pol/ board and six major subreddits (AskReddit, politics, conspiracy, The_Donald, news, worldnews).

Key Insights and Ideas

  • Anonymity Breeds Contagion: Unsurprisingly, comments posted anonymously contained a significantly higher proportion of hate speech (10.7%) compared to comments from registered accounts (7.6%).
  • The "Power User" Problem: A small percentage of highly active users generate the vast majority of toxic content. Casual commenters rarely post hate speech, whereas "power users" are much more likely to frequently disseminate hateful remarks.
  • Linguistic Triggers: Articles that attract high levels of hate speech (dubbed "HATE articles") possess distinct linguistic fingerprints. They score high in Clout (authoritative, confident language meant to influence) and low in Tone (indicating high negativity). They also rely heavily on first- and third-person singular pronouns (I, she, he) and express high levels of anger and anxiety.
  • The Cross-Platform Echo Chamber: When an article is shared on 4chan or Reddit, it acts as a beacon. Hateful comments on the original news site spike dramatically within two hours of the external post, suggesting coordinated or organic "raids" from these fringe communities.

Key Results

  • Site-by-Site Toxicity: Infowars (an alt-right conspiracy site) and Times of Israel shared the highest percentage of hate comments at 15.3%. Conversely, the New York Times maintained the lowest at just 0.9%, highlighting the impact of strict moderation or differing audience demographics.
Percentage of hate comments across different news websites.
  • The Partisanship Paradox: While extreme-right sites attract the highest raw volume of comments per article, center and left-leaning sites actually attract a higher fraction of hate speech relative to their total comment volume.
CDF of the number of comments (a) and the fraction of hate comments (b) per article, grouped by political partisanship.
  • Event Synchronization: Changepoint analysis successfully matched major spikes in hate speech to highly polarizing events, including the 2016 US Presidential Debates, Trump's Inauguration, Brexit protests, and the "Unite the Right" rally in Charlottesville.
Temporal overview showing significant changepoints in overall comments (a) and hate comments (b) aligned with major real-world events.
  • Sticky Toxicity: Once a news article receives a substantial burst of hate comments (often post-4chan/Reddit sharing), the fraction of hateful comments remains elevated for a long period, showing that toxicity has a long half-life.

Limitations and Future Work

A primary limitation is that the dataset was collected post-hoc. Because news sites actively moderate content, an unknown percentage of the most severe hate speech had likely already been deleted before the crawlers could capture it. Additionally, automated tools like the Perspective API can still miss nuanced, sarcastic, or highly coded hate speech.

For future work, the authors plan to develop proactive classification systems that can predict which news articles are highly vulnerable to being "raided" before the toxicity spills over, allowing publishers to implement defensive moderation strategies ahead of time.

Summary updated

Abstract

The Web has become the main source for news acquisition. At the same time, news discussion has become more social: users can post comments on news articles or discuss news articles on other platforms like Reddit. These features empower and enable discussions among the users; however, they also act as the medium for the dissemination of toxic discourse and hate speech. The research community lacks a general understanding on what type of content attracts hateful discourse and the possible effects of social networks on the commenting activity on news articles. In this work, we perform a large-scale quantitative analysis of 125M comments posted on 412K news articles over the course of 19 months. We analyze the content of the collected articles and their comments using temporal analysis, user-based analysis, and linguistic analysis, to shed light on what elements attract hateful comments on news articles. We also investigate commenting activity when an article is posted on either 4chan's Politically Incorrect board (/pol/) or six selected subreddits. We find statistically significant increases in hateful commenting activity around real-world divisive events like the "Unite the Right" rally in Charlottesville and political events like the second and third 2016 US presidential debates. Also, we find that articles that attract a substantial number of hateful comments have different linguistic characteristics when compared to articles that do not attract hateful comments. Furthermore, we observe that the post of a news articles on either /pol/ or the six subreddits is correlated with an increase of (hateful) commenting activity on the news articles.
BibTeX
@misc{zannettou2020measuring,
  author = {Savvas Zannettou and Mai ElSherief and Elizabeth Belding and Shirin Nilizadeh and Gianluca Stringhini},
  title = {{Measuring and Characterizing Hate Speech on News Websites}},
  year = {2020},
  eprint = {2005.07926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.07926},
}

Loading related content...

Figure