TL;DR
This paper investigates how repeated interactions with Large Language Models (LLMs) shape human trust, finding that users start with high baseline trust that only diminishes when they receive explicit, ground-truth feedback on the AI's errors. Without such feedback, people remain highly susceptible to persuasion—even by consistently wrong models—suggesting a significant risk of overreliance in domains where the truth is difficult to verify.
What Problem Does This Paper Address?
As conversational AI becomes a staple in professional and personal decision-making, a critical question emerges: How well do we calibrate our trust to an AI’s actual reliability?
Most existing research focuses on one-off interactions with anonymous AI agents. However, real-world usage is often longitudinal. If an AI assistant helps you correctly four times in a row, you likely trust it more on the fifth. But what if it was wrong those four times? The authors explore whether humans can detect unreliability through "indirect cues" (like circular reasoning or rhetorical pressure) or if we are effectively "blind" to AI incompetence without external verification. They also test the "truth advantage"—the idea that it is inherently easier to persuade someone of a true statement than a false one.
Methods
The researchers conducted four pre-registered experiments using a Judge-Advisor System (JAS) paradigm. Participants first answered 30 difficult reasoning questions (e.g., physics, logic, probability) and rated their confidence. They then entered an "advice phase" for five of those questions, interacting with an advisor to update their answers.
Key variables manipulated across the experiments included:
- Helpfulness: The advisor was either 100% accurate (Helpful), 0% accurate (Unhelpful), or 50% accurate (Mixed) during the initial "intro" trials.
- Feedback: Whether participants were shown the correct answer after each trial.
- Interaction Mode: Multi-turn natural language conversation vs. static text hints.
- Advisor Type: LLM (GPT-4.5 or GPT-5) vs. human participants.
The primary metrics were Persuader Success (PS)—the rate at which participants adopted the advisor's position—and Confidence Change (CC).
Datasets
The study utilized a custom-curated set of 30 two-alternative forced-choice (2AFC) reasoning questions. These were specifically designed to have objective ground truths but low baseline accuracy (~47%) and confidence (~64%) among participants, ensuring there was "room" for persuasion. The questions covered physics (e.g., the "canoe and rock" buoyancy problem), logic, and probability (e.g., the Monty Hall problem).
Key Insights and Ideas
- Trust is Lost, Not Gained: Contrary to the idea that we slowly build trust in AI, participants started with high reliance. Trust didn't increase much with a helpful track record; instead, it stayed high and only dropped when the AI was proven wrong.
- The Feedback Vacuum: Without explicit ground-truth feedback, participants were unable to calibrate. They were just as likely to be persuaded by a model that had been wrong four times as one that had been right four times.
- LLMs Out-Persuade Humans: LLMs were significantly more persuasive than human advisors (83% vs 65% success rate). Interestingly, LLMs benefited more from being "helpful" than humans did, perhaps because humans struggled to explain even correct answers convincingly, whereas LLMs provided information-dense (if sometimes hallucinated) rationales.
- The Fragility of Truth: The "truth advantage" is not a given. It only emerged when participants were already confident in their initial (correct) position and engaged in a dialogue. For participants with low confidence or those receiving static hints, arguing for a lie was just as effective as arguing for the truth.
- Flattery as a Feature: Qualitative analysis revealed that "flattering the target" (e.g., "That's a common and very logical thought, but...") was one of the most effective strategies for persuader success.
Key Results
- Experiment 1 (With Feedback): Participants successfully calibrated. Helpful models had a 95% success rate, while Unhelpful models dropped to 61%.
- Experiment 2 (No Feedback): Calibration failed completely. Helpful models (90% PS) and Unhelpful models (89% PS) were virtually indistinguishable to participants.
- Experiment 3 (Hints vs. Conversation): Multi-turn conversation was far more persuasive than static hints (42% vs 17% PS).
- Experiment 4 (LLM vs. Human): LLMs were more persuasive than humans across the board.
Limitations and Future Work
The authors acknowledge several constraints:
- Short-term Interaction: The study only looked at five trials in a single session. Real-world trust dynamics over months or years might differ.
- Incentives: Participants were not financially incentivized for accuracy, which might increase "gullibility."
- Domain Specificity: The results focus on fact-based reasoning. Trust calibration might look very different for subjective, political, or identity-relevant topics.
- Model Selection: The study used specific versions of GPT-4.5 and GPT-5; different architectures might exhibit different persuasive "styles."
Future research should investigate whether users can learn to calibrate using indirect feedback (e.g., the AI contradicting itself) and how trust evolves in high-stakes environments where errors have significant consequences.
Summary updated