The PGNSC Benchmark: How Do We Predict Where Information Spreads?

Alexander Taylor, Wei WangACL 2024 · OpenReview

Abstract

Social networks have become ideal vehicles for news dissemination because posted content is easily able to reach users beyond a news outlet’s direct audience. Understanding how information is transmitted among communities of users is a critical step towards understanding the impact social networks have on real-world events. Two significant barriers in this vein of work are identifying user clusters and meaningfully characterizing these communities. Thus, we propose the PGNSC benchmark, which builds information pathways based on the audiences of influential news sources and uses their content to characterize the communities. We present methods of aggregating these news-source-centric communities and for constructing the community feature representations that are used sequentially to construct information pathway prediction pipelines. Lastly, we perform extensive experiments to demonstrate the performance of baseline pipeline constructions and to highlight the possibilities for future work.

Introduction

Predicting the trajectory of information through social networks is a fundamental challenge in computational social science. While individual-level prediction is often hampered by sparse data and inconsistent user behavior, aggregating users into communities offers a more robust path forward. However, defining these communities and representing their collective interests remains difficult. Traditional benchmarks often overlook the multimodal nature of modern news—combining text and imagery—and fail to capture the dynamic shifts in community engagement over time.

Social media interaction graph showing various engagement types (Reply, Retweet, Quote Retweet) that form the basis of the raw data.

The PGNSC Benchmark

To address these gaps, researchers at UCLA developed the PGNSC (Prominent Global News Sources for Covid-19) benchmark. This dataset spans 15 countries and 150 news organizations, encompassing over 300,000 tweets and 34,000 news articles collected during the pandemic. Unlike previous datasets, PGNSC provides a multimodal view, including article text, titles, and images, mapped to a community-level graph with over 100 million edges.

Methodology

The proposed framework for information pathway prediction follows a three-stage sequential pipeline:

1. Community Aggregation

The core innovation lies in grouping users around influential news sources. The researchers evaluated several network heuristics for this mapping, such as Engagement, Interaction (PageRank-based), and Influence-Passivity (IP). The latter balances a node's influence on others against its own receptivity to information.

The community mapping process: transitioning from individual user interactions to news-source-centric community pathways.

2. Feature Initialization

To characterize these communities, the framework uses Large Language Models (LLMs) to process the content they consume. The study compared single-stage encoding (e.g., Longformer) against multi-stage pipelines that first summarize content (using models like PRIMERA, Llama-2, or GPT-4) before encoding the "essence" of the news.

Finally, a Heterogeneous Temporal Graph Neural Network (HTGNN) is employed to predict the propagation of information between these news-centric communities over time.

Key Findings

The experimental results yielded several critical insights:

  • The Power of Summarization: Two-stage pipelines that summarized content before encoding outperformed single-stage methods by an average of 5.4 points. This suggests that distilling news content into its core themes is more effective for community characterization than processing raw, noisy text.
  • Heuristic Performance: The Influence-Passivity (IP) heuristic emerged as the most effective method for community mapping, leading to significant performance gains (6.55 to 10.98 points in GMAUC) over simpler baselines.
  • Multimodal Limitations: Surprisingly, incorporating images via CLIP encodings slightly decreased performance compared to text-only BERT encodings. This indicates that while images drive human engagement, they may not yet provide a sufficiently unique signal for differentiating news communities in current models.
  • LLM Efficiency: Open-source models like Llama-2 showed parity with, and in some cases slightly exceeded, the performance of GPT-4 for the specific task of news summarization in this context.

Conclusion and Future Work

The PGNSC benchmark demonstrates that community-level modeling, particularly when centered on news sources and enhanced by LLM-based summarization, significantly improves the predictability of information flow. Future research will focus on improving the scalability of network heuristics to billion-node graphs and refining multimodal integration to better leverage the informational content of news imagery.

Summary updated

Abstract

Social networks have become ideal vehicles for news dissemination because posted content is easily able to reach users beyond a news outlet’s direct audience. Understanding how information is transmitted among communities of users is a critical step towards understanding the impact social networks have on real-world events. Two significant barriers in this vein of work are identifying user clusters and meaningfully characterizing these communities. Thus, we propose the PGNSC benchmark, which builds information pathways based on the audiences of influential news sources and uses their content to characterize the communities. We present methods of aggregating these news-source-centric communities and for constructing the community feature representations that are used sequentially to construct information pathway prediction pipelines. Lastly, we perform extensive experiments to demonstrate the performance of baseline pipeline constructions and to highlight the possibilities for future work.
BibTeX
@inproceedings{taylor2024pgnsc,
  author = {Alexander Taylor and Wei Wang},
  title = {{The PGNSC Benchmark: How Do We Predict Where Information Spreads?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=zz7AaXrdc5},
}
Figure