WildFrame: Comparing Framing in Humans and LLMs on Naturally Occurring Texts

Gili Lior, Liron Nacchace, Gabriel StanovskyarXiv 2025 · DOI · arXiv

Abstract

The framing effect is a well-documented cognitive bias where the presentation of information significantly alters its perception. This paper introduces WILDFRAME, the first dataset designed to compare human and Large Language Model (LLM) responses to framing using naturally occurring text. By analyzing 1,000 statements from Amazon reviews, the study finds that LLMs exhibit human-like susceptibility to framing, particularly showing a "positivity asymmetry" where positive framing of negative facts is more persuasive than the reverse. Notably, the study highlights varying degrees of alignment between specific model families and human behavior, with some advanced models showing unexpected divergence from human patterns.

Introduction

Cognitive biases, such as the framing effect (e.g., preferring "90% fat-free" over "10% fat"), are inherent in human communication and decision-making. As LLMs are increasingly integrated into complex reasoning tasks, understanding whether they mirror these human biases is crucial for both safety and utility. Previous research has largely relied on synthetic, "lab-grown" data that lacks the nuance of real-world language. This study addresses this gap by creating the WILDFRAME benchmark to evaluate 11 different LLMs against human consensus on naturally occurring text.

Methodology

The researchers developed a three-step pipeline to generate and validate the WILDFRAME dataset, ensuring the use of "natural" rather than purely synthetic language.

The WILDFRAME pipeline: (a) Extracting base statements, (b) Applying opposite framing, and (c) Sentiment analysis annotation via crowdsourcing.

The process involved:

  1. Extraction: 1,000 real-world statements (500 positive, 500 negative) were pulled from Amazon reviews using the SPIKE tool.
  2. Reframing: GPT-4 was used to add a "framing clause" that introduced the opposite sentiment (e.g., adding a negative consequence to a positive achievement).
  3. Annotation: Sentiment shifts were recorded via crowdsourcing (Amazon Mechanical Turk) and a controlled in-house group to establish a human baseline.

Results and Discussion

The Positivity Asymmetry

A key finding of the study is the "positivity asymmetry" shared by both humans and LLMs. Both groups are significantly more swayed by positive framing of negative statements than negative framing of positive ones. It appears easier to make a "bad" thing sound acceptable than it is to ruin the perception of something inherently "good."

Human-Model Alignment and the GPT Paradox

All tested LLMs showed a significant correlation with human sentiment shifts, with Pearson’s $r \ge 0.52$. However, the degree of alignment varied significantly across model families.

Pearson Correlation Coefficient of various LLMs with human behavior regarding sentiment shifts.

In a surprising twist termed the "GPT Paradox," GPT-3.5 and GPT-4o showed the lowest correlation with human behavior regarding framing. While these are high-performing models, they seem to process linguistic nuances differently than humans, potentially due to their specific alignment training. In contrast, GPT-5-mini achieved a correlation of $r = 0.61$, and larger models in the Llama and Mistral families generally tracked closer to human consensus as they scaled. Interestingly, the Gemma family showed an inverse relationship, with the smaller 9b model aligning more closely with humans than the 27b version.

Robustness and Variance

LLM predictions remained consistent approximately 70% of the time even when framing was paraphrased, suggesting that their susceptibility to framing is a stable characteristic. However, human judgments showed much higher variance (0.137) than LLMs (0.108), highlighting that while LLMs mimic human patterns, they lack the diverse subjectivity found in human populations.

Conclusion

The WILDFRAME benchmark demonstrates that LLMs are not objective observers but inherit human-like framing biases. While they mimic the patterns of human sentiment shifts, the divergence in alignment among top-tier models like GPT-4o suggests that "smarter" models do not necessarily become more human-like in their cognitive processing. Future work should explore these effects in high-stakes domains like legal reasoning or medical advice to determine the implications of these biases in professional contexts.

Summary updated

BibTeX
@misc{lior2025wildframe,
  author = {Gili Lior and Liron Nacchace and Gabriel Stanovsky},
  title = {{WildFrame: Comparing Framing in Humans and LLMs on Naturally Occurring Texts}},
  year = {2025},
  eprint = {2502.17091},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.17091},
}
Figure