Evaluating Large Language Models for Belief Inference: Mapping Belief Networks at Scale

Trisevgeni Papakonstantinou, Antonina Zhiteneva, Ana Yutong Ma, Derek Powell, Zachary HorneEMNLP 2025

TL;DR

This paper introduces a novel pipeline using a fine-tuned GPT-40 model to automatically infer complex belief structures from large-scale social media data. The approach demonstrates that LLMs can effectively map how individual beliefs co-occur and cohere, offering a scalable alternative to traditional survey methods for understanding public opinion.

What Problem Does This Paper Address?

Understanding how people's beliefs are interconnected is crucial for various fields, from education to public health. Traditionally, this has been studied using surveys and Likert scales, which are time-consuming and limited to small, controlled datasets. The rise of social media platforms like Reddit offers a vast, unstructured source of public discourse, but extracting coherent belief structures from this noisy data at scale has been a significant methodological challenge. Previous work has focused on belief change or argumentation, but not on systematically mapping the underlying belief networks themselves. This paper aims to bridge this gap by developing and validating a method to infer these belief structures directly from naturalistic online text.

Methods

The core of this work is an automated analytical pipeline that leverages Large Language Models (LLMs) for belief inference. The pipeline operates in two main phases:

  • Phase 1: Belief Extraction: This phase focuses on identifying explicit belief statements within user posts. A fine-tuned GPT-40 model is used to extract these statements, preserving their original wording as much as possible. This is crucial for maintaining the author's intent and avoiding bias.
  • Phase 2: Theory Inference: Here, the extracted belief statements are mapped onto broader, pre-specified belief categories (derived from Pew Research surveys). A fine-tuned GPT-40 model is then trained to predict the likelihood of a user endorsing these broader beliefs based on their extracted statements, using a Likert scale (1-7) to mirror human survey responses.

The authors also employed a two-stage filtering process for their data, initially using keyword searches and then refining with GPT-40 to ensure relevance and reduce noise. For evaluation, they compared a baseline GPT-40, a fine-tuned GPT-40, and an OpenAI o3-mini model against human-annotated ground truth.

Datasets

  • Change My View (CMV) Dataset: This dataset, scraped from the r/ChangeMyView subreddit, forms the primary source of naturalistic text data. Posts from June 2005 to June 2024 were collected.
  • Pew Research Center Surveys: Data and question frameworks from Pew Research surveys were used to define the set of 64 belief topics and broader belief categories for the analysis.
  • Human Annotated Data: A subset of 200 CMV posts, covering 20 distinct belief topics, was independently annotated by two human annotators to create a "ground truth" dataset for evaluating the LLM's performance.
  • U.S. Respondent Survey Data: For correlation analysis, the authors compared their inferred belief correlations with those computed from a survey of 376 U.S. adults.

Key Insights and Ideas

  • LLMs as Belief Extractors: The study demonstrates that fine-tuned LLMs, specifically GPT-40, can effectively extract explicit belief statements from noisy social media text, a task previously requiring intensive human annotation.
  • Scalability of Belief Inference: The pipeline offers a significant leap in scalability, enabling the analysis of belief structures across vast amounts of online data, which is infeasible with traditional survey methods.
  • Generalization to Novel Beliefs: The fine-tuned GPT-40 model showed remarkable ability to infer beliefs on topics it wasn't explicitly trained on, suggesting its potential for real-world deployment in diverse contexts.
  • Alignment with Human Belief Structures: The inferred belief correlations from social media data showed a moderate agreement with correlations derived from human survey data, indicating that the LLM-based approach captures meaningful patterns of belief co-occurrence.
  • Ideological Clustering: The analysis of belief correlations revealed distinct clusters that align with known ideological patterns, suggesting the method can uncover underlying societal divisions.

Key Results

  • The fine-tuned GPT-40 model achieved the best performance across all evaluation metrics, including ±1 Accuracy, Cohen's κ, Spearman's ρ, and Cosine Similarity, often performing at a level comparable to human annotators.
  • Specifically, the fine-tuned GPT-40 achieved ±1 Accuracy of 82.22% (compared to human annotators at 80.68%) and a Cosine Similarity of 0.93 (vs. human at 0.91) in the main evaluation.
  • When tested on unseen belief topics (cross-validation), the fine-tuned GPT-40 maintained strong performance (±1 Accuracy = 86.67%, Cosine Similarity = 0.96).
  • The comparison of correlation matrices between CMV data and survey data yielded a Pearson correlation coefficient of r = 0.56, indicating moderate agreement.

Limitations and Future Work

  • Representativeness of Data: The CMV subreddit users are not representative of the general population, potentially skewing the observed beliefs and co-occurrence patterns towards a younger, middle-class, American male demographic. Generalizing these findings requires caution.
  • Sparsity of Belief Signals: Users often post about a limited subset of topics, meaning their full belief structures are only partially observable, leading to incomplete inferred relationships.
  • Model Evaluation Scope: The evaluation was limited to OpenAI models. Exploring other LLM architectures or fine-tuning strategies could yield different results.
  • Inference Circularity: While efforts were made to mitigate this, the use of GPT-40 in both filtering and inference stages warrants careful consideration.
  • Qualitative Alignment: The comparison with survey data was primarily qualitative. Future work could involve more rigorous quantitative comparisons, accounting for sample size and data uncertainty.

The authors suggest that their pipeline is a proof of concept and can be applied to explore belief structures in novel contexts, offering a powerful tool for tracing belief networks from real-world text at scale.

Summary updated

BibTeX
@inproceedings{papakonstantinou2025evaluating,
  author = {Trisevgeni Papakonstantinou and Antonina Zhiteneva and Ana Yutong Ma and Derek Powell and Zachary Horne},
  title = {{Evaluating Large Language Models for Belief Inference: Mapping Belief Networks at Scale}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure