DeSMOG: Detecting Stance in Media On Global Warming

Yiwei Luo, Dallas Card, Dan Jurafsky

Introduction: The Art of Opinion-Framing

Traditional stance detection in Natural Language Processing (NLP) typically focuses on a binary classification: is a text "for" or "against" a specific topic? However, researchers at Stanford argue that this approach overlooks a sophisticated persuasive strategy prevalent in climate discourse: opinion-framing. This technique involves ascribing an opinion to a specific source to either validate one's own position (affirming) or undermine an opponent's (doubting).

For example, a climate activist might state, "Leading scientists agree that global warming is serious," while a skeptic might counter, "Mistaken scientists claim that global warming is serious." Despite the identical core proposition—that global warming is serious—the framing completely alters the intended message. The DeSMOG project introduces the Global Warming Stance Dataset (GWSD) to study these nuanced layers of argumentation.

The Mechanics of Opinion-Framing

To analyze how media outlets use these linguistic tricks, the researchers developed a pipeline to extract "triples" from text. These triples consist of a SOURCE (the entity being quoted), a PREDICATE (the verb used to attribute the thought), and the OPINION (the embedded clause containing the stance).

Components of an opinion triple: Source, Predicate, and Opinion.

Methodology: From Triples to Stance

The study utilized a multi-stage computational approach:

  1. Extraction: Using dependency parsing and coreference resolution, the team identified triples across a massive corpus of 56,000 news articles from 63 U.S. sources (2000–2020).
  2. Stance Classification: A BERT-based classifier was trained to determine if an opinion agrees, disagrees, or is neutral toward the target: "Climate change/global warming is a serious concern."
  3. Weighted BERT: Recognizing that human annotators often disagree on stance, the model was trained on the probability distribution of labels from multiple annotators rather than a single "gold" label.
  4. Lexicon Building: The researchers categorized predicates (e.g., concede vs. insist) and modifiers (e.g., peer-reviewed vs. misleading) into "affirming" and "doubting" devices.

Strategic Asymmetry and "Hypocritical Attribution"

One of the paper's most striking findings is the prevalence of unfaithful attribution. Approximately 37-38% of opinions in the dataset were ascribed to sources in a way that contradicted their well-known public stances. This often takes the form of "hypocritical attribution," where an outlet frames an opponent as secretly agreeing with the author's view. For instance, left-leaning media might highlight that "Exxon knew fossil fuels caused global warming," framing the corporation as a hypocrite that maintains a public facade while possessing private "truth."

Asymmetry in Media Tactics

While both global-warming-accepting and global-warming-skeptical media use similar linguistic tools (like using reveal to affirm and pretend to doubt), there is a significant quantitative asymmetry in how these tools are deployed.

Distribution of self-affirming vs. opponent-doubting framing across media slants.

As shown in the data, left-leaning (LL) media focuses heavily on "self-affirming" discourse—quoting sources that support the consensus. In contrast, right-leaning (RL) media engages in significantly more "opponent-doubting" discourse, frequently quoting climate-consensus sources only to frame their claims as dubious or fraudulent.

The Subjectivity of Truth

The research also highlights that stance is often in the eye of the beholder. Using Bayesian hierarchical ordinal regression, the authors found that an annotator’s own political affiliation significantly influenced their perception. Self-identified Republicans were statistically more likely to label a sentence as "agreeing" with climate urgency than non-Republicans, suggesting that personal priors shift the baseline for what constitutes a "serious" statement.

Conclusion and Future Directions

The BERT classifier achieved 75% accuracy, outperforming linear baselines and proving competitive with human performance. However, the study acknowledges limitations, such as the use of binary "left/right" media labels and the lack of distinction between op-eds and hard news. Future work aims to distinguish between different types of sources—such as scientists, corporations, and politicians—to see how framing strategies vary across different actors in the climate debate. Ultimately, DeSMOG emphasizes that NLP researchers must account for demographic biases in data labeling, as these biases can bake "subjective truths" into model training.

Summary updated

Abstract

Citing opinions is a powerful yet understudied strategy in argumentation. For example, an environmental activist might say, "Leading scientists agree that global warming is a serious concern," framing a clause which affirms their own stance ("that global warming is serious") as an opinion endorsed ("[scientists] agree") by a reputable source ("leading"). In contrast, a global warming denier might frame the same clause as the opinion of an untrustworthy source with a predicate connoting doubt: "Mistaken scientists claim [...]." Our work studies opinion-framing in the global warming (GW) debate, an increasingly partisan issue that has received little attention in NLP. We introduce Global Warming Stance Dataset (GWSD), a dataset of stance-labeled GW sentences, and train a BERT classifier to study novel aspects of argumentation in how different sides of a debate represent their own and each other's opinions. From 56K news articles, we find that similar linguistic devices for self-affirming and opponent-doubting discourse are used across GW-accepting and skeptic media, though GW-skeptical media shows more opponent-doubt. We also find that authors often characterize sources as hypocritical, by ascribing opinions expressing the author's own view to source entities known to publicly endorse the opposing view. We release our stance dataset, model, and lexicons of framing devices for future work on opinion-framing and the automatic detection of GW stance.
BibTeX
@inproceedings{luo2020desmog,
  author = {Yiwei Luo and Dallas Card and Dan Jurafsky},
  title = {{DeSMOG: Detecting Stance in Media On Global Warming}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.296},
}
Figure