Abstract
Large Language Models (LLMs) are increasingly used in conversational Question Answering (QA) systems. While they exhibit high fluency and sophisticated linguistic grounding, there is a growing concern regarding their factual accuracy. This study investigates the "unjustified trust" paradox: users are more likely to trust and prefer AI responses that mirror their own language (lexical alignment), even when those responses are factually incorrect. By analyzing linguistic behaviors and introducing the K-F1++ metric, the authors demonstrate how surface-level alignment can mask hallucinations, leading users to favor "stochastic parrots" over more faithful models.
Introduction: The Grounding Paradox
In human dialogue, "grounding" behaviors—such as repeating key phrases or using pronouns—signal mutual understanding. As LLMs like GPT-3 become more adept at these behaviors, they create a compelling illusion of competence. However, a critical gap exists between fluency (how natural a response sounds) and faithfulness (how accurately it reflects source material). This paper explores whether these conversational "vibes" act as a "trust hack," causing users to overlook factual errors.
Methodology
The researchers evaluated three model configurations on the TopiOCQA dataset, which features complex, multi-turn information-seeking dialogues:
- DPR+FiD: A fine-tuned "closed" baseline specifically trained for the dataset.
- GPT-3 (Closed-book): Relying solely on internal knowledge.
- DPR+GPT-3 (Open-book): Augmented with relevant Wikipedia snippets.
Evaluation combined automatic metrics with human studies. A key contribution is K-F1++, a metric that discounts words repeated from the user's question to isolate the new information provided by the model. This prevents models from inflating their scores through mere parroting.
Key Insights: The "Vibe" Over Substance
The study revealed several troubling trends in how users interact with LLMs:
- The Parrotting Preference: GPT-3 variants tend to be more verbose and exhibit higher lexical alignment than fine-tuned models. Humans consistently preferred this behavior, even when the content was unfaithful.
- The Trust Hack: High lexical alignment (repeating the user) boosted trust scores to 58%, compared to 32% for low alignment.
- Syntactic Bias: Users favored full, short sentences over the fragments common in natural human speech, associating formality with reliability.
Key Results and the Faithfulness Gap
The experimental results quantify the disparity between user perception and model accuracy:
- Faithfulness Scores: The fine-tuned DPR+FiD model achieved 57.8% faithfulness, while closed-book GPT-3 plummeted to 22.0%.
- Over-Alignment: GPT-3 showed a lexical alignment "Recall" of 38%, significantly higher than the human gold standard of 10.8%, suggesting that LLMs "over-align" to please users.
- Preference for Hallucination: In the preference task, users chose the open-book GPT-3 model 70% of the time over the baseline. Alarmingly, 48% of these preferred responses were unfaithful (hallucinated).
Conclusion and Future Directions
The findings suggest that the conversational prowess of modern LLMs can be deceptive. Users are easily misled by "stochastic parrots" that prioritize linguistic alignment over factual grounding. Future work must focus on developing better detection methods for "extrinsic hallucinations"—plausible-sounding but false claims—and designing user interfaces that alert users when a model's confidence is not backed by evidence. The community is urged to move beyond surface-level metrics and prioritize faithfulness in conversational AI development.
Summary updated