Do People Trust AI Assistants Even When They Are Consistently Wrong?

Cameron R. Jones, Julia Wong, Aaron S. Cohen, James Woolley, Benjamin K. BergenarXiv 2026 · arXiv

TL;DR

This paper investigates how repeated interactions with Large Language Models (LLMs) shape human trust, finding that users start with high baseline trust that only diminishes when they receive explicit, ground-truth feedback on the AI's errors. Without such feedback, people remain highly susceptible to persuasion—even by consistently wrong models—suggesting a significant risk of overreliance in domains where the truth is difficult to verify.

What Problem Does This Paper Address?

As conversational AI becomes a staple in professional and personal decision-making, a critical question emerges: How well do we calibrate our trust to an AI’s actual reliability?

Most existing research focuses on one-off interactions with anonymous AI agents. However, real-world usage is often longitudinal. If an AI assistant helps you correctly four times in a row, you likely trust it more on the fifth. But what if it was wrong those four times? The authors explore whether humans can detect unreliability through "indirect cues" (like circular reasoning or rhetorical pressure) or if we are effectively "blind" to AI incompetence without external verification. They also test the "truth advantage"—the idea that it is inherently easier to persuade someone of a true statement than a false one.

Methods

The researchers conducted four pre-registered experiments using a Judge-Advisor System (JAS) paradigm. Participants first answered 30 difficult reasoning questions (e.g., physics, logic, probability) and rated their confidence. They then entered an "advice phase" for five of those questions, interacting with an advisor to update their answers.

The experimental procedure involved a survey, item selection based on confidence, advice trials with an LLM or human, and an exit survey.

Key variables manipulated across the experiments included:

  • Helpfulness: The advisor was either 100% accurate (Helpful), 0% accurate (Unhelpful), or 50% accurate (Mixed) during the initial "intro" trials.
  • Feedback: Whether participants were shown the correct answer after each trial.
  • Interaction Mode: Multi-turn natural language conversation vs. static text hints.
  • Advisor Type: LLM (GPT-4.5 or GPT-5) vs. human participants.

The primary metrics were Persuader Success (PS)—the rate at which participants adopted the advisor's position—and Confidence Change (CC).

Datasets

The study utilized a custom-curated set of 30 two-alternative forced-choice (2AFC) reasoning questions. These were specifically designed to have objective ground truths but low baseline accuracy (~47%) and confidence (~64%) among participants, ensuring there was "room" for persuasion. The questions covered physics (e.g., the "canoe and rock" buoyancy problem), logic, and probability (e.g., the Monty Hall problem).

Key Insights and Ideas

  • Trust is Lost, Not Gained: Contrary to the idea that we slowly build trust in AI, participants started with high reliance. Trust didn't increase much with a helpful track record; instead, it stayed high and only dropped when the AI was proven wrong.
  • The Feedback Vacuum: Without explicit ground-truth feedback, participants were unable to calibrate. They were just as likely to be persuaded by a model that had been wrong four times as one that had been right four times.
  • LLMs Out-Persuade Humans: LLMs were significantly more persuasive than human advisors (83% vs 65% success rate). Interestingly, LLMs benefited more from being "helpful" than humans did, perhaps because humans struggled to explain even correct answers convincingly, whereas LLMs provided information-dense (if sometimes hallucinated) rationales.
  • The Fragility of Truth: The "truth advantage" is not a given. It only emerged when participants were already confident in their initial (correct) position and engaged in a dialogue. For participants with low confidence or those receiving static hints, arguing for a lie was just as effective as arguing for the truth.
  • Flattery as a Feature: Qualitative analysis revealed that "flattering the target" (e.g., "That's a common and very logical thought, but...") was one of the most effective strategies for persuader success.

Key Results

  • Experiment 1 (With Feedback): Participants successfully calibrated. Helpful models had a 95% success rate, while Unhelpful models dropped to 61%.
  • Experiment 2 (No Feedback): Calibration failed completely. Helpful models (90% PS) and Unhelpful models (89% PS) were virtually indistinguishable to participants.
Results from Experiment 1 (with feedback) and Experiment 2 (no feedback) show that calibration only occurs when ground truth is provided.
  • Experiment 3 (Hints vs. Conversation): Multi-turn conversation was far more persuasive than static hints (42% vs 17% PS).
  • Experiment 4 (LLM vs. Human): LLMs were more persuasive than humans across the board.
LLMs achieved higher persuader success than humans and showed a stronger truth advantage in certain conditions.

Limitations and Future Work

The authors acknowledge several constraints:

  1. Short-term Interaction: The study only looked at five trials in a single session. Real-world trust dynamics over months or years might differ.
  2. Incentives: Participants were not financially incentivized for accuracy, which might increase "gullibility."
  3. Domain Specificity: The results focus on fact-based reasoning. Trust calibration might look very different for subjective, political, or identity-relevant topics.
  4. Model Selection: The study used specific versions of GPT-4.5 and GPT-5; different architectures might exhibit different persuasive "styles."

Future research should investigate whether users can learn to calibrate using indirect feedback (e.g., the AI contradicting itself) and how trust evolves in high-stakes environments where errors have significant consequences.

Summary updated

Abstract

People are increasingly using conversational AI systems to make decisions despite concerns around their reliability. How well do people calibrate to AI systems' accuracy in multi-turn conversations? We measure how people's reliance on LLMs' advice changes across repeated interactions, varying whether the AI assistant is helpful (100% accurate), unhelpful (0% accurate), or mixed (50% accurate). Across 4 experiments, we find people learn to trust helpful models more than unhelpful ones (67% vs 21% of the time), but only when they receive external feedback on the models' accuracy. Without feedback, people continue to be persuaded 48% of the time by a consistently unhelpful model (vs 38% for helpful). Rather than gaining trust in helpful models, participant reliance on LLM advisors tended to be high in early trials and only fell with evidence of inaccuracy. Participants trusted LLM advisors more than they trusted human advisors (83% vs 65%, Experiment 4), and LLMs benefited more from having a helpful track record than humans did. In contrast with previous work, we find limited evidence of an advantage for arguing for the truth (versus a falsehood). Truth advantages emerged only when participants were already very confident in their position, and did not emerge at all for static hints. Collectively, the results suggest that people may be swayed by LLM advice, even when it is consistently false, unless they have access to timely ground-truth feedback. Unfortunately, in many domains such feedback may not be available, leading to overreliance on AI.
BibTeX
@misc{jones2026do,
  author = {Cameron R. Jones and Julia Wong and Aaron S. Cohen and James Woolley and Benjamin K. Bergen},
  title = {{Do People Trust AI Assistants Even When They Are Consistently Wrong?}},
  year = {2026},
  eprint = {2611.14833},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14833},
}
Figure