Large Language Models respond to Influence like Humans

Lewis Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly Mai, Maria Do Mar Vau, Matthew Caldwell, Augustine Mavor-ParkerACL 2023

TL;DR

This paper investigates whether Large Language Models (LLMs) can be used to model how humans change their beliefs and attitudes due to external influence. Through two studies, the researchers found that LLMs, like humans, are susceptible to phenomena like the Illusory Truth Effect and populist framing, suggesting their potential as models for studying psychological influence.

What Problem Does This Paper Address?

Understanding how influence affects human psychology is crucial for applications ranging from combating disinformation to promoting healthy behaviors. However, studying this directly with human participants is slow, expensive, and ethically challenging. This paper proposes using LLMs as a more efficient and ethical alternative, akin to how animal models are used in pharmacology. The core question is whether LLMs, beyond exhibiting static psychological traits, can also model the dynamic process of psychological change in response to influence.

Methods

The researchers conducted two main studies:

  • Illusory Truth Effect (ITE): This study tested whether LLMs exhibit the ITE, a phenomenon where repeated exposure to a statement increases its perceived truthfulness, even without evidence. They designed an experiment where both human and LLM participants were exposed to statements and then rated their truthfulness. They also explored whether this effect extended to other attributes like interest, sentiment, and importance, and whether it occurred with same-attribute exposure versus mixed-attribute exposure.
  • Populist Framing of News (PFN): This study investigated whether LLMs are influenced by populist framing in news articles, which emphasizes in-group vs. out-group divisions. They adapted a previous study that exposed human participants to different versions of a news article (neutral, anti-elite, anti-immigrant, or both) and measured persuasion and mobilization. For the LLM study, they simulated participant demographics and relative deprivation levels and then used LLM completion to generate responses to probe statements about persuasion and mobilization.

Datasets

  • ITE Study: The researchers created 100 novel statements for the ITE experiment. They collected data from 1000 simulated LLM participants and 1000 human participants.
  • PFN Study: The PFN study utilized data and methodology from a previous study by Bos et al. (2020) which involved 7286 human participants across 15 countries. For the LLM portion, they simulated demographic information and relative deprivation scores based on the summary statistics from the original study. They collected data from 2153 simulated LLM participants.

Key Insights and Ideas

  • LLMs as Dynamic Influence Models: The most significant insight is that LLMs can indeed model dynamic psychological change in response to influence, not just static traits. This opens up new avenues for research in social psychology and persuasion.
  • ITE Replication in LLMs: LLMs exhibit the Illusory Truth Effect similarly to humans, showing that mere exposure can boost perceived truthfulness, even when the exposure attribute differs from the rating attribute.
  • Populist Framing Effects Replicated (and Surprising Ones): LLMs replicated some surprising findings from human studies on populist framing, such as a counter-intuitive decrease in mobilization when immigrants were blamed. This suggests LLMs can capture nuanced and unexpected human responses to persuasive framing.
  • LLMs are Less Variable (but More Variable Per-Statement): While LLM responses generally showed less variability than human responses in the PFN study, the per-statement ITE effect was more variable for LLMs, indicating a potential difference in how individual statements are processed.

Key Results

  • ITE Study:
    • Both humans and LLMs showed a significant Illusory Truth Effect (ITE) for truth ratings when exposed to statements with different attributes.
  • This effect was not observed for other attributes like interest, sentiment, or importance, confirming the ITE is specific to truth.
  • Same-attribute exposure did not significantly boost ratings for either humans or LLMs.
  • While the overall pattern of effects was similar, the per-statement ITE was more variable for LLMs than for humans.
  • PFN Study:
    • LLMs showed similar persuasion scores to humans, but higher mobilization scores with less variability.
  • Both humans and LLMs were influenced by anti-elite framing (increasing persuasion).
  • Both humans and LLMs showed a surprising decrease in mobilization when anti-immigrant framing was used.
  • LLMs showed a positive effect of anti-elite framing on persuasion, which was not significant in humans.
  • LLMs showed a positive interaction between relative deprivation and populist framing on mobilization, which was significant in humans but not in LLMs.

Limitations and Future Work

  • ITE Study Limitations: The authors acknowledge that the selection of testing statements was "irreproducible" and suggest this should be addressed in future work. They also note that the lack of simulated covariance between participant traits in the PFN study might explain some failures to replicate human results.
  • PFN Study Limitations: The LLM study was limited by the monthly API cap, resulting in fewer participants (2153) than the original human study (7286). The failure to replicate the interaction between relative deprivation and populist framing on mobilization is noted as disappointing.
  • Future Work: The authors suggest that LLMs have significant potential for modeling influence, even without specific adaptation. Future work should focus on addressing the limitations mentioned, potentially by exploring more sophisticated prompting strategies or larger-scale LLM simulations. The potential for LLMs to be used for malign applications like strategic influence underscores the importance of this research.

Summary updated

BibTeX
@inproceedings{griffin2023large,
  author = {Lewis Griffin and Bennett Kleinberg and Maximilian Mozes and Kimberly Mai and Maria Do Mar Vau and Matthew Caldwell and Augustine Mavor-Parker},
  title = {{Large Language Models respond to Influence like Humans}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure