A Murder and Protests, the Capitol Riot, and the Chauvin Trial: Estimating Disparate News Media Stance

Sujan Dutta, Beibei Li, Daniel S. Nagin, Ashiqur R. KhudaBukhshIJCAI 2022 · DOI

Introduction

The polarization of US cable news is a well-documented phenomenon, yet quantifying this bias objectively and at scale remains a significant challenge. Traditional sentiment analysis often misses the nuances of political "stance"—the underlying ideological position a speaker holds toward a specific subject. This study utilizes advanced Natural Language Processing (NLP) and a novel active learning pipeline to measure how major networks (CNN, MSNBC, and Fox News) shifted their stances on policing during the social upheavals of 2020 and 2021. By analyzing thousands of transcripts, researchers have quantified how media narratives diverge in response to major national events.

Methodology: Stance as Natural Language Inference

The researchers framed stance mining as a Natural Language Inference (NLI) task. In this framework, a model determines if a "premise" (a sentence from a news transcript) entails, contradicts, or remains neutral toward a "hypothesis." To quantify these stances, the authors introduced the entailment ratio: the fraction of sentences in a corpus that logically support a specific pro-police hypothesis, such as "Police protect us" or "Police make us safe."

Active Learning and Inconsistency Sampling

Because off-the-shelf NLI models often struggle with domain-specific topics like policing, the authors developed an active learning pipeline. This included a novel strategy called inconsistency sampling. This method identifies instances where a model provides logically conflicting predictions for semantically similar hypotheses—for example, predicting that a sentence entails "Police protect us" but contradicts "Police make us safe." By prioritizing these "confused" instances for human annotation, the researchers significantly improved model performance, achieving a Macro F1 score of 75.42, a vast improvement over baseline models.

Data and Corpora

The study analyzed a substantial corpus of 34,710 news transcripts from CNN, MSNBC, and Fox News, spanning May 2019 to July 2021. From this, they extracted specific "policing corpora" containing the token "police," resulting in nearly 120,000 relevant sentences. This dataset allowed for a granular look at how language varies across networks.

Qualitative examples of news transcripts from MSNBC, CNN, and Fox News showing differing stances on policing.

Key Results: The Great Divergence and the Capitol Flip

The study's findings reveal that media coverage responds rapidly to the partisan preferences of its viewership, creating distinct versions of reality for different audiences.

Baseline and the "Great Divergence"

Before the murder of George Floyd in May 2020 ($\mathcal{T}_{pre-Floyd}$), all three networks maintained remarkably similar entailment ratios (approximately 0.2). This suggests a relatively unified baseline for policing discourse prior to the period of intense social unrest. However, the aftermath of the murder triggered a "Great Divergence." During the Floyd protests ($\mathcal{T}_{Floyd}$), entailment ratios for CNN and MSNBC dropped by 36.5% and 45.2% respectively, while Fox News’s ratio increased by 21.3%.

The Capitol Riot Narrative Shift

The January 6th Capitol Riot ($\mathcal{T}_{capitol}$) prompted another dramatic shift. During this period, MSNBC’s entailment ratio for the "police protect us" hypothesis surged by 145.9%, as the narrative shifted to portray officers as defenders of democratic institutions against rioters. Interestingly, during the trial of Derek Chauvin ($\mathcal{T}_{trial}$), Fox News's entailment ratio reached its peak, significantly higher than its competitors.

Entailment ratios for a pro-police hypothesis across major news events, illustrating the divergence in media stance.

Conclusion

The data suggests that cable news networks do more than report the news; they curate worldviews that align with their audience's existing biases. This "siloed" reporting makes bridging partisan divides increasingly difficult. While large language models carry inherent biases from their training data, this methodology provides a robust, scalable framework for mapping media disparity across other polarizing topics such as climate change, reproductive rights, or immigration.

Summary updated

BibTeX
@inproceedings{dutta2022murder,
  author = {Sujan Dutta and Beibei Li and Daniel S. Nagin and Ashiqur R. KhudaBukhsh},
  title = {{A Murder and Protests, the Capitol Riot, and the Chauvin Trial: Estimating Disparate News Media Stance}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/702},
}
Figure