Abstract
The spread of misinformation on social media platforms threatens democratic processes, contributes to massive economic losses, and endangers public health. Many efforts to address misinformation focus on a knowledge deficit model and propose interventions for improving users’ critical thinking through access to facts. Such efforts are often hampered by challenges with scalability, and by platform users’ confirmation bias. The emergence of generative AI presents promising opportunities for countering misinformation at scale across ideological barriers. In this paper, we present (1) an experiment with a simulated social media environment to measure effectiveness of misinformation interventions generated by large language models (LLMs), (2) a second experiment with personalized explanations tailored to the demographics and beliefs of users with the goal of alleviating confirmation bias, and (3) an analysis of potential harms posed by personalized generative AI when exploited for automated creation of disinformation. Our findings confirm that LLM-based interventions are highly effective at correcting user behavior (improving overall user accuracy at reliability labeling up to 47.6%). Furthermore, we find that users favor more personalized interventions when making decisions about news reliability.
Introduction
The digital information ecosystem is currently struggling to contain the spread of misinformation. Traditional interventions, such as manual fact-check tags or "disputed" labels, face two significant hurdles: scalability and confirmation bias. Professional fact-checking organizations cannot keep pace with the sheer volume of social media content. Moreover, even when a fact-check is available, users often ignore it if it contradicts their existing worldviews. This research investigates whether generative AI can provide a middle ground: scalable, automated interventions that are personalized to a user’s demographics and beliefs to effectively "nudge" them toward the truth.
Methodology
The researchers conducted a three-phase study using a simulated social media environment mimicking platforms like Facebook or X.
- Phase I (Non-personalized Interventions): An A/B test comparing five types of interventions, ranging from simple labels to explanations generated by GPT-4 and "reaction frames" (templated explanations of a writer's intent).
- Phase II (Personalized Interventions): The team developed a pipeline to infer user attributes (gender, race, age, education, and politics) using survey questions from the Pew Research American Trends Panel. They then prompted GPT-4 to generate explanations for news veracity specifically tailored to those inferred attributes.
- Phase III (Malicious Use Case): To test the "dark side" of this technology, GPT-4 was instructed to generate disinformation headlines for common conspiracy theories (e.g., "Flat Earth," "White Genocide") tailored to specific demographic groups to see if personalization made the falsehoods more believable.
Key Insights and Results
The Personalization Premium
Users do not just want facts; they want facts that resonate with their identity. The study found that when an AI’s explanation was "aligned" with a user’s identity (e.g., using specific linguistic markers or addressing cultural concerns), the user rated the intervention as significantly more helpful. On a 4-point scale, personalized explanations received a mean helpfulness score of 2.90, compared to 2.71 for non-personalized ones.
Accuracy and Linguistic Shifts
LLM-based interventions proved highly effective. In Phase I, GPT-4 explanations helped users reach 98.19% accuracy in identifying news reliability, a dramatic increase from a baseline of roughly 51% without interventions. Personalization also significantly altered the text's characteristics: tailoring for "educated" users increased formality by 18.46% and decreased readability (increasing complexity), while tailoring for "Black" users led to less formal language usage.
The Double-Edged Sword
The same mechanism that allows an LLM to debunk a lie for a specific person allows it to craft a lie for that same person. Users were remarkably poor at identifying GPT-4-generated disinformation. While they identified real news correctly 82.32% of the time, they only correctly labeled GPT-4-generated disinformation as false in 35.84% of cases. This suggests that bad actors could use LLMs to conduct micro-targeted influence operations at an unprecedented scale.
Discussion and Limitations
While generative AI offers a powerful tool for mitigation, several critical caveats remain. First, these interventions depend entirely on "ground truth"; if an AI model is confidently wrong about a fact, its personalized explanation will only make that hallucination more persuasive. Second, inferring user attributes like race or religion for fact-checking presents significant privacy concerns.
Future work should explore how to provide these benefits without violating user trust or data regulations. The authors suggest that the most effective path forward involves a "human-in-the-loop" system, where AI scales the delivery of facts verified by professional human fact-checkers.
Summary updated