An Interactive Framework for Profiling News Media Sources

Nikhil Mehta, Dan Goldwasser

TL;DR

This paper introduces an interactive framework that combines graph-based models, large language models (LLMs), and human input to effectively profile news media sources for bias and factuality, especially in challenging "emerging events" scenarios. The approach significantly improves detection accuracy with minimal human effort, demonstrating the power of human-AI collaboration.

What Problem Does This Paper Address?

The proliferation of social media has led to a rapid spread of fake and biased news, making it crucial to identify and profile the sources behind this content. While automated systems exist, they struggle, particularly in "emerging events" where new sources and content appear that weren't in the training data. Manually profiling every source is also infeasible due to the sheer volume. This paper tackles the challenge of accurately and efficiently profiling news media sources, especially in dynamic, unseen environments, by leveraging human insight to guide AI models.

Methods

The core of the proposed framework is an iterative process to build "information communities" of users with similar perspectives.

  1. Graph Model Initialization: A pre-trained graph neural network (R-GCN) is used to generate initial embeddings for users, articles, and sources. These embeddings capture relationships within the social media network.
  2. Candidate Community Formation: K-means clustering is applied to user embeddings to form initial "candidate" information communities.
  3. User Summarization (LLM): A large language model (GPT-3) is used to generate concise textual summaries of each user's content and perspective, based on their profile and tweets.
  4. Human Validation: A human interactor reviews the LLM-generated summaries to validate and refine the candidate communities, ensuring they are cohesive and represent similar viewpoints. This is a crucial step where human judgment excels over LLMs.
  5. Community Expansion (LLM + Graph): The validated communities are then expanded. The graph model identifies potential new members based on embeddings, and GPT-3, prompted with examples from human-validated communities, helps assign users to these communities. This step leverages LLMs' ability to learn from few-shot examples.
  6. Iterative Refinement: The process of community formation, validation, and expansion is repeated iteratively.
  7. Unsupervised Graph Training: New edges are added to the graph connecting users within the same validated community. The graph model is then fine-tuned using link prediction on this enriched graph, without requiring new labeled data, to better capture user relationships and improve source profiling.

Datasets

  • Media Bias/Fact Check dataset: Used for training and evaluation of news source factuality and bias.
  • Twitter Data: Scraped for specific events (Black Lives Matter, Abortion/Feminism) to construct the graph and evaluate the framework in emerging event settings. This data includes articles, user interactions, and source information.

Key Insights and Ideas

  • Human-AI Synergy is Key: The framework demonstrates that combining human judgment with AI (graph models and LLMs) significantly outperforms purely automated approaches, especially in complex, dynamic scenarios. Humans excel at identifying nuanced user similarity, which LLMs struggle with.
  • Information Communities as a Proxy: The core idea is that users within cohesive "information communities" (sharing similar perspectives) are more likely to interact with and share content from sources with similar bias and factuality. Profiling these communities indirectly profiles the sources.
  • Minimal Human Interaction is Powerful: The framework achieves significant performance gains with a remarkably small number of human interactions (as few as 1-2 per event), making it practical for real-world deployment.
  • Emerging Events are a Crucial Testbed: Evaluating on "emerging events" (inductive setting, unseen data) highlights the framework's robustness and ability to adapt to novel situations, a key weakness of traditional methods.
  • LLMs as Amplifiers, Not Sole Decision-Makers: LLMs are effectively used to summarize and assist in expansion, but human validation remains critical for initial community formation and ensuring accuracy.

Key Results

  • Significant Performance Improvements: The interactive framework (LLM + Humans) achieved substantial improvements over baselines (Graph Only, LLM Only) on both fake news and bias source detection.
  • ~33% improvement on fake news source macro F1.
  • ~40% improvement on bias news source macro F1.
  • Minimal Interaction, Maximum Impact: The best performance was achieved with only 1-2 human interactions, creating a small number of validated communities that were then expanded.
  • Cohesive Communities: Analysis showed that the formed communities, especially with human input, were largely cohesive, with users sharing similar bias labels. The LLM-only approach showed less cohesiveness.

Limitations and Future Work

  • Domain Specificity: The current work focuses on English-language Twitter data and specific events (BLM, Abortion/Feminism). Generalizing to other domains, languages, and platforms is a future direction.
  • LLM Transparency: The reliance on proprietary LLMs like GPT-3, whose internal workings are not fully public, is a potential concern.
  • Human Interactor Bias: While the interaction task is simple, the potential for biased human interactors in real-world deployments needs careful consideration. Using multiple experts and majority voting is suggested.
  • Scalability of Human Interaction: While minimal interactions are effective, exploring more complex interaction setups with multiple experts or more nuanced tasks could be beneficial.
  • Source Label Inconsistencies: The study acknowledges that source labels from external datasets might not always be perfectly accurate for all content, but hypothesizes that the framework's interaction process can help correct for such inaccuracies.
  • Computational Resources: Experiments were run on a single GPU, but scaling to larger datasets might require more resources.

Summary updated

Abstract

The recent rise of social media has led to the spread of large amounts of fake and biased news, content published with the intent to sway beliefs. While detecting and profiling the sources that spread this news is important to maintain a healthy society, it is challenging for automated systems. In this paper, we propose an interactive framework for news media profiling. It combines the strengths of graph based news media profiling models, Pre-trained Large Language Models, and human insight to characterize the social context on social media. Experimental results show that with as little as 5 human interactions, our framework can rapidly detect fake and biased news media, even in the most challenging settings of emerging news events, where test data is unseen.
BibTeX
@inproceedings{mehta2024interactive,
  author = {Nikhil Mehta and Dan Goldwasser},
  title = {{An Interactive Framework for Profiling News Media Sources}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.3},
}
Figure