Large Scale Narrative Messaging around Climate Change: A Cross-Cultural Comparison

Haiqi Zhou, David G Hobson, Derek Ruths, Andrew PiperACL 2024

TL;DR

This paper explores how large language models (LLMs) like GPT-4 can be used to automatically extract the underlying "moral" or lesson from news articles about climate change. By comparing articles from North America and China, they found distinct cultural differences in the narratives used, highlighting the potential of LLMs for understanding global climate communication.

What Problem Does This Paper Address?

Understanding how we communicate about complex issues like climate change is crucial for driving action. Stories are a powerful tool for conveying information and shaping understanding, but manually analyzing the vast amount of climate-related news is a monumental task. This paper addresses the challenge of automatically identifying the implicit "narrative messaging" – the core moral or lesson – within these stories, especially across different cultural contexts. Existing methods for analyzing narratives are often manual and limited in scale. The authors aim to bridge this gap by leveraging LLMs to uncover these deeper messages and understand how they vary culturally.

Methods

The core of this research is the use of Large Language Models (LLMs), specifically GPT-4, to extract narrative features from news articles. They define "narrative messaging" as the intrinsic moral or lesson of a story. To achieve this, they developed a prompting sequence for GPT-4. This sequence involves first summarizing the article, then identifying key elements like the protagonist, antagonist, central topic, and overall valence. Crucially, it also prompts the LLM to extract a "moral" or lesson, and associated positive and negative keywords.

They experimented with two prompt frameworks: a full-context pipeline (where all previous prompts and responses are fed back into the model) and a simplified framework (using only the summary as context). The simplified framework was chosen for its efficiency.

For validation, they employed a combination of human evaluation and automated metrics. Undergraduate students and paid crowd workers (via Amazon Mechanical Turk) were used to assess the LLM's outputs. They also used semantic textual similarity (STS) scores (ROUGE, GloVe, etc.) to compare the LLM's extracted morals with human-generated ones.

Datasets

The study utilized a dataset of approximately 15,000 news articles published in 2023, sourced from Dow Jones Factiva. These articles were filtered by length (250-2,500 words) and selected using keywords related to environmental issues: "climate change," "pollution," "carbon emissions," "renewable energy," and "sustainability." The dataset was segmented by language into Mandarin and English.

For validation, a smaller test dataset of 64 news articles was used, drawn from sources including CNN, Al-Jazeera English, and four Chinese-language news outlets.

Key Insights and Ideas

  • LLMs as Narrative Extractors: The research demonstrates that LLMs can effectively surface latent narrative morals, which are often implicit and not explicitly stated in news articles.
  • Cultural Divergence in Climate Narratives: A significant finding is the distinct difference in narrative messaging between North American and Chinese media. North American sources tend to focus on individualistic and crisis-driven themes, while Chinese sources emphasize developmental and cooperative narratives.
  • "Moral" as a Lens for Understanding: Shifting the focus from "what happened" to "why is this being told?" (i.e., the moral) provides a powerful way to understand collective belief systems and cultural values shaping discourse.
  • Validation of LLM Performance: The study provides robust validation, showing that GPT-4's extracted morals are highly comparable to human judgments, both through direct agreement and semantic similarity metrics.

Key Results

  • Valence Differences: North American sources were associated with a lower average valence (M=2.87) compared to Chinese-speaking regions (M=3.45). Mainland Chinese sources were more positive than offshore Chinese sources.
  • Distinctive Moral Keywords:
    • Chinese sources emphasized "international cooperation" and "sustainable development."
    • North American sources focused on "addressing" and "adapting" to climate change.
  • Within North America, Conservative outlets highlighted "markets," "investments," and "economic/financial issues," often framing climate change through an "adapting" lens. Liberal outlets leaned towards "addressing" and a more holistic view.
  • Chinese state media framed "sustainable" as a modifier of development, while offshore Chinese media focused on individual-level terms like "community" and "recycling."
  • LLM Agreement: Human evaluators showed majority agreement with GPT-4's extracted morals in 75-90% of cases for the simplified prompt framework. Automated metrics also showed high similarity between human and GPT-generated morals.

Limitations and Future Work

  • Cultural Sampling: The study sampled eight news outlets across two cultural/ideological contexts. Future work should expand to include more cultures and a wider range of news sources.
  • Keyword Filtering: The dataset was filtered by specific keywords, which might have excluded relevant climate change discussions. Exploring alternative methods for identifying a fuller sample is suggested.
  • LLM Biases: While LLMs are powerful, understanding their implicit biases and norms is crucial.
  • Replication Variability: Despite setting temperature to zero, some variability in LLM outputs was observed, making exact replication challenging.
  • Model Dependence: The reliance on GPT-4 as the primary LLM is a limitation. Future research should explore other large frontier models and potentially smaller, specialized models to reduce carbon footprint.
  • Further Exploration: The authors suggest applying different clustering methods, comparing LLM-based coding with existing methodologies, and using LLMs to answer previously human-coded questions for deeper insights into narrative messaging.

Summary updated

BibTeX
@inproceedings{zhou2024large,
  author = {Haiqi Zhou and David G Hobson and Derek Ruths and Andrew Piper},
  title = {{Large Scale Narrative Messaging around Climate Change: A Cross-Cultural Comparison}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure