Granular Analysis of Social Media Users’ Truthfulness Stances Toward Climate Change Factual Claims

Haiqi Zhang, Zhengyuan Zhu, Zeyu Zhang, Jacob Devasier, Chengkai LiACL 2024

TL;DR

This paper introduces a novel framework for analyzing how social media users perceive the truthfulness of climate change claims. By leveraging LLMs to build a detailed taxonomy of climate change topics and then performing stance detection on tweets, they found that users tend to believe claims are true regardless of their veracity, especially in political and economic discussions. This highlights a critical need for improved media literacy and critical thinking skills.

What Problem Does This Paper Address?

The urgency of climate change necessitates understanding public discourse surrounding it. Social media platforms are a major source of information and discussion, but they also present challenges in discerning factual information from misinformation. While previous research has explored climate change discourse through topic modeling and sentiment analysis, there's a need for a more granular understanding of how users evaluate the truthfulness of specific factual claims across various climate change sub-topics. This paper aims to bridge this gap by developing a framework to analyze users' truthfulness stances towards factual claims within a structured taxonomy of climate change topics.

Methods

The authors propose a four-step framework:

  1. Factual Claim Collection: They gathered factual claims related to climate change from five reputable fact-checking websites (PolitiFact, Snopes, Full Fact, Metafact, AP News). Claims were identified using keywords from the EPA's climate change glossary. Verdicts were standardized into "Truth," "Uncertain," and "Misinformation."
  2. Tweet Collection: For each factual claim, they collected corresponding tweets from X (formerly Twitter) to create claim-tweet pairs. This was done by tokenizing claims and using noun, verb, and adjective tokens as search queries.
  3. Taxonomy Construction: A hierarchical taxonomy of climate change topics was automatically constructed using a Large Language Model (LLM), specifically Zephyr, with a human-in-the-loop approach. This involved multiple rounds of LLM generation and human refinement to create broad, medium, and detailed topic categories.
  4. Truthfulness Stance Detection: A supervised fine-tuning approach was used to train an LLM (Zephyr) to classify the stance of a tweet towards a factual claim. Stances were categorized as positive (believing the claim is true), negative (believing it's false), or neutral/no stance.

Datasets

  • Factual Claims: 1,409 unique climate change-related factual claims collected from five fact-checking websites, spanning from November 2007 to May 2024.
  • Tweets: 13,050 tweets collected for 729 of the factual claims.
  • Annotated Dataset for Stance Detection: An in-house dataset of 1,871 high-quality stance annotations for claim-tweet pairs, not limited to climate change topics, used for supervised fine-tuning.

Key Insights and Ideas

  • Ubiquitous Belief in Truth: A significant finding is that social media users tend to believe factual claims are true, regardless of their actual veracity. This suggests a general tendency towards accepting information presented as fact.
  • Lack of Discernment in Key Areas: Users exhibit a notable lack of discernment between facts and misinformation, particularly in discussions related to politics, economy, and the environment. This is a critical area for concern.
  • Topic-Specific Judgment Variation: The accuracy of users' judgments varies significantly across different climate change topics. Some topics show higher accuracy in stance detection, while others reveal a pronounced struggle.
  • LLMs for Taxonomy and Stance Detection: The paper demonstrates the effectiveness of LLMs, augmented with human oversight, for both creating detailed topic taxonomies and performing truthfulness stance detection.

Key Results

  • Taxonomy Performance: The automatically generated taxonomy achieved an average accuracy of 83% for broad topics and 62.5% for medium topics when evaluated by human annotators.
  • Stance Detection Performance: The truthfulness stance detection classifier achieved an average precision of 0.837, recall of 0.808, and macro F1 score of 0.821 on the test set.
  • Stance Distribution: Out of 13,050 tweets analyzed, the majority (61.33%) expressed a positive stance (believing the claim is true).
  • Inaccurate Topics: Topics like "Elections" (Politics), "Agriculture" (Environment), and "Energy Prices" (Economy) showed particularly low accuracy and macro F1 scores (e.g., 0.117 for Elections), indicating poor public judgment in these areas.
  • Accurate Topics: Topics such as "Others" under "Government Policies" and "Climate Change Impacts" under "Climate Science" showed higher accuracy.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that the LLM sometimes generates inconsistent topics, necessitating human-in-the-loop refinement.
  • The taxonomy construction involved a "human-in-the-loop" process, which can be resource-intensive.
  • The paper notes that the "Others" category was excluded from some analyses due to small sample sizes.
  • Claims with "Uncertain" verdicts were also excluded from some analyses.
  • Future Work: The paper suggests that the developed framework can be extended to analyze public judgments across multi-level topics related to climate change. The findings highlight the need for targeted interventions, such as improved critical thinking education and robust fact-checking, to enhance public discernment and the accuracy of information on social media.

Summary updated

BibTeX
@inproceedings{zhang2024granular,
  author = {Haiqi Zhang and Zhengyuan Zhu and Zeyu Zhang and Jacob Devasier and Chengkai Li},
  title = {{Granular Analysis of Social Media Users’ Truthfulness Stances Toward Climate Change Factual Claims}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure