From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models

Julia Mendelsohn, Ronan Le Bras, Yejin Choi, Maarten Sap

This paper presents the first large-scale computational study of "dogwhistles"—coded expressions that convey a benign literal meaning to the public while signaling provocative or hateful messages to a specific in-group. By contributing a comprehensive glossary of over 300 dogwhistles, the authors demonstrate that while large language models (LLMs) like GPT-3 have a limited ability to decode these terms, dogwhistles remain highly effective at evading state-of-the-art toxicity detection systems.

The Problem of Coded Rhetoric

In the realm of online safety and political discourse, most research focuses on overt hate speech. However, bad actors often use dogwhistles to bypass both social repercussions and algorithmic content moderation. For example, a politician might use the term "cosmopolitan" to signal "Jewish" to an anti-Semitic base while maintaining "plausible deniability" by claiming they simply meant "worldly."

Conceptual model of a dogwhistle communication, illustrating how a speaker uses a message to signal a hidden persona to an in-group while maintaining a literal meaning for the out-group.

The challenge for Natural Language Processing (NLP) is three-fold:

  1. Context Dependency: Meaning is derived from the speaker's persona and the audience's specialized knowledge.
  2. Evolution: Dogwhistles evolve rapidly to stay ahead of public awareness.
  3. Invisibility: Out-group members often fail to recognize a word is being used as a slur.

Methodology and Taxonomy

The researchers developed a multi-pronged approach to formalize the study of these "bullhorns in disguise":

  • Taxonomy Development: Dogwhistles were categorized by Register (formal/offline vs. informal/online), Type (Type I: signaling a persona; Type II: adding hidden meaning), and Persona (the specific in-group identity being signaled).
  • Glossary Curation: A dataset of 340 English-language dogwhistles and over 1,000 surface forms was built from academic literature, political speeches, and community-sourced wikis.
  • LLM Probing: The team tested GPT-3’s ability to "surface" (list) dogwhistles and identify their hidden meanings. GPT-3 identified covert meanings for ~80% of the glossary in at least one prompt variation but struggled heavily with transphobic terms and informal, internet-born codes (12.9% recall for informal vs. 69.4% for formal).

Toxicity Evasion and the "Bullhorn" Vulnerability

The most alarming insight from the study is that dogwhistles effectively "blind" toxicity detectors. Using the Perspective API, the researchers scored sentences where slurs or standard group labels were replaced by dogwhistles within hateful templates (derived from the HateCheck suite).

Perspective API scores for dogwhistles vs. slurs and standard terms, showing how dogwhistles significantly lower toxicity and identity attack ratings.

As shown in the experimental results, replacing a standard group label (like "Jewish") with a dogwhistle (like "cosmopolitan") causes toxicity scores to plummet. This confirms that dogwhistles are a viable and dangerous strategy for evading current moderation systems, even though the harm to the target group remains identical.

The Lifecycle of a Dogwhistle

Historical analysis of terms like "Willie Horton" suggests that once a dogwhistle is successfully "called out" and becomes widely recognized by the out-group, it loses its effectiveness. However, the "plausible deniability shield" remains a powerful tool for speakers to retreat to literal meanings when challenged, making automated bans difficult to justify.

Future Directions

Because dogwhistles are dynamic and evolve to avoid detection, static glossaries will eventually become obsolete. The authors suggest that future research must move toward "context-aware" detection that analyzes the speaker's persona and the surrounding discourse rather than relying on keyword-based systems. Furthermore, while this study focused on the U.S. context, dogwhistles are a global phenomenon requiring cross-lingual and cross-cultural investigation.

Summary updated

Abstract

Dogwhistles are coded expressions that simultaneously convey one meaning to a broad audience and a second, often hateful or provocative, meaning to a narrow in-group; they are deployed to evade both political repercussions and algorithmic content moderation. For example, the word “cosmopolitan” in a sentence such as “we need to end the cosmopolitan experiment” can mean “worldly” to many but also secretly mean “Jewish” to a select few. We present the first large-scale computational investigation of dogwhistles. We develop a typology of dogwhistles, curate the largest-to-date glossary of over 300 dogwhistles with rich contextual information and examples, and analyze their usage in historical U.S. politicians’ speeches. We then assess whether a large language model (GPT-3) can identify dogwhistles and their meanings, and find that GPT-3’s performance varies widely across types of dogwhistles and targeted groups. Finally, we show that harmful content containing dogwhistles avoids toxicity detection, highlighting online risks presented by such coded language. This work sheds light on the theoretical and applied importance of dogwhistles in both NLP and computational social science, and provides resources to facilitate future research in modeling dogwhistles and mitigating their online harms.
BibTeX
@inproceedings{mendelsohn2023from,
  author = {Julia Mendelsohn and Ronan Le Bras and Yejin Choi and Maarten Sap},
  title = {{From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.845},
}
Figure