DeSMOG: Detecting Stance in Media On Global Warming

Yiwei Luo, Dallas Card, Dan Jurafsky

Abstract

Persuasion in the global warming debate often relies on subtle "opinion-framing"—the strategic choice of sources and verbs to attribute claims. This paper introduces the Global Warming Stance Dataset (GWSD) and a BERT-based classifier that identifies media stance with human-level accuracy. Analysis reveals that while both sides use similar linguistic tools, right-leaning media engages in significantly more "opponent-doubting" discourse.

Introduction: Beyond Simple Stance Detection

Traditional Natural Language Processing (NLP) stance detection focuses on whether a sentence is "for" or "against" a topic. However, in polarized debates like climate change, authors often use attribution to frame facts. There is a significant rhetorical difference between stating "Scientists show temperatures are rising" (affirming) and "Scientists claim temperatures are rising" (doubting). The former validates the statement, while the latter invites skepticism. This paper addresses the gap in handling such opinion-framing by providing a large-scale dataset and a robust classification pipeline.

Methodology and Data

The researchers analyzed 56,000 news articles using a multi-stage pipeline. First, they used syntactic dependency parsing to extract "triples" consisting of a SOURCE, a PREDICATE, and the OPINION.

Structure of an opinion triple: Source, Predicate, and Opinion.

The core datasets include:

  • GWSD (Global Warming Stance Dataset): 2,050 opinion spans with 16,400 human annotations.
  • Op_full: 500,000 opinions extracted from 56,000 articles published between 2000 and 2020.
  • Lexicons: Curated lists of "affirming" (e.g., prove, understand) and "doubting" (e.g., claim, allege) framing devices.

To handle subjective labeling, the authors used a Bayesian hierarchical model to weight the BERT classifier's loss function based on annotator agreement, rather than simple majority voting.

Analysis of Media Framing

The study identifies a "hypocrisy" strategy where outlets attribute opinions to their opponents to highlight contradictions (e.g., "Exxon knew fossil fuels caused warming"). Interestingly, the linguistic "weapons" are symmetric: if the Left uses "scientists know," the Right uses "skeptics know."

However, a key finding is Asymmetric Skepticism. While both sides use similar tools for self-affirmation, right-leaning (RL) media uses significantly more "opponent-doubting" discourse compared to left-leaning (LL) media.

Proportions of self-affirming vs. opponent-doubting coverage in left-leaning (LL) and right-leaning (RL) media.

Key Results

  • Model Performance: The BERT classifier achieved 75% accuracy, slightly exceeding estimated human performance (71%) and outperforming linear baselines.
  • Unfaithful Attribution: Approximately 37-38% of opinions were "unfaithfully ascribed," meaning the attributed opinion contradicted the source's known public position—often used to frame an opponent as hypocritical.
  • Annotator Bias: Annotators' own political affiliations influenced their labels; Republicans were 1.05x more likely to label sentences as "agreeing" with the seriousness of global warming, potentially because they perceived the language as more alarmist or exaggerated.

Limitations and Future Directions

The study's binary categorization of media (Left vs. Right) may overlook nuances in centrist reporting. Additionally, the pipeline does not yet distinguish between straight news and op-eds, where framing is typically more aggressive. Future work aims to apply this framing analysis to other scientific debates, such as COVID-19 policies or vaccine efficacy.

Summary updated

BibTeX
@inproceedings{luo2020desmog,
  author = {Yiwei Luo and Dallas Card and Dan Jurafsky},
  title = {{DeSMOG: Detecting Stance in Media On Global Warming}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.296},
}
Figure