The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering

Sabrina Chiesurin, Dimitris Dimakopoulos, Marco Antonio Sobrevilla Cabezudo, Arash Eshghi, Ioannis Papaioannou, Verena Rieser, Ioannis Konstas

Abstract

Large Language Models (LLMs) are increasingly used in conversational Question Answering (QA) systems. While they exhibit high fluency and sophisticated linguistic grounding, there is a growing concern regarding their factual accuracy. This study investigates the "unjustified trust" paradox: users are more likely to trust and prefer AI responses that mirror their own language (lexical alignment), even when those responses are factually incorrect. By analyzing linguistic behaviors and introducing the K-F1++ metric, the authors demonstrate how surface-level alignment can mask hallucinations, leading users to favor "stochastic parrots" over more faithful models.

Introduction: The Grounding Paradox

In human dialogue, "grounding" behaviors—such as repeating key phrases or using pronouns—signal mutual understanding. As LLMs like GPT-3 become more adept at these behaviors, they create a compelling illusion of competence. However, a critical gap exists between fluency (how natural a response sounds) and faithfulness (how accurately it reflects source material). This paper explores whether these conversational "vibes" act as a "trust hack," causing users to overlook factual errors.

Examples of linguistic phenomena and their impact on user preference vs. faithfulness.

Methodology

The researchers evaluated three model configurations on the TopiOCQA dataset, which features complex, multi-turn information-seeking dialogues:

  1. DPR+FiD: A fine-tuned "closed" baseline specifically trained for the dataset.
  2. GPT-3 (Closed-book): Relying solely on internal knowledge.
  3. DPR+GPT-3 (Open-book): Augmented with relevant Wikipedia snippets.

Evaluation combined automatic metrics with human studies. A key contribution is K-F1++, a metric that discounts words repeated from the user's question to isolate the new information provided by the model. This prevents models from inflating their scores through mere parroting.

Key Insights: The "Vibe" Over Substance

The study revealed several troubling trends in how users interact with LLMs:

  • The Parrotting Preference: GPT-3 variants tend to be more verbose and exhibit higher lexical alignment than fine-tuned models. Humans consistently preferred this behavior, even when the content was unfaithful.
  • The Trust Hack: High lexical alignment (repeating the user) boosted trust scores to 58%, compared to 32% for low alignment.
  • Syntactic Bias: Users favored full, short sentences over the fragments common in natural human speech, associating formality with reliability.
User preference comparison across models, highlighting the high preference for unfaithful GPT-3 responses.

Key Results and the Faithfulness Gap

The experimental results quantify the disparity between user perception and model accuracy:

  • Faithfulness Scores: The fine-tuned DPR+FiD model achieved 57.8% faithfulness, while closed-book GPT-3 plummeted to 22.0%.
  • Over-Alignment: GPT-3 showed a lexical alignment "Recall" of 38%, significantly higher than the human gold standard of 10.8%, suggesting that LLMs "over-align" to please users.
  • Preference for Hallucination: In the preference task, users chose the open-book GPT-3 model 70% of the time over the baseline. Alarmingly, 48% of these preferred responses were unfaithful (hallucinated).

Conclusion and Future Directions

The findings suggest that the conversational prowess of modern LLMs can be deceptive. Users are easily misled by "stochastic parrots" that prioritize linguistic alignment over factual grounding. Future work must focus on developing better detection methods for "extrinsic hallucinations"—plausible-sounding but false claims—and designing user interfaces that alert users when a model's confidence is not backed by evidence. The community is urged to move beyond surface-level metrics and prioritize faithfulness in conversational AI development.

Summary updated

Abstract

Large language models are known to produce output which sounds fluent and convincing, but is also often wrong, e.g. "unfaithful" with respect to a rationale as retrieved from a knowledge base. In this paper, we show that task-based systems which exhibit certain advanced linguistic dialog behaviors, such as lexical alignment (repeating what the user said), are in fact preferred and trusted more, whereas other phenomena, such as pronouns and ellipsis are dis-preferred. We use open-domain question answering systems as our test-bed for task based dialog generation and compare several open- and closed-book models. Our results highlight the danger of systems that appear to be trustworthy by parroting user input while providing an unfaithful response.
BibTeX
@inproceedings{chiesurin2023dangers,
  author = {Sabrina Chiesurin and Dimitris Dimakopoulos and Marco Antonio Sobrevilla Cabezudo and Arash Eshghi and Ioannis Papaioannou and Verena Rieser and Ioannis Konstas},
  title = {{The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.60},
}
Figure