TL;DR
Large language models (LLMs) can produce fluent but factually incorrect responses ("hallucinations"). This paper reveals a dangerous paradox: users tend to trust LLMs more when they exhibit conversational behaviors like repeating user input (lexical alignment), even if the information is wrong. This suggests that superficial linguistic cues can override accuracy, leading to unjustified trust.
What Problem Does This Paper Address?
The paper tackles the growing issue of "hallucinations" in large language models (LLMs), particularly in conversational question-answering (QA) systems. While LLMs are adept at generating human-like text, they often produce factually incorrect statements that are not grounded in their knowledge sources. This unfaithfulness, combined with their fluency, creates a deceptive user experience. The core problem is understanding how users perceive and trust these systems, especially when certain linguistic features might mislead them into believing the system is more reliable than it is. The authors specifically investigate how conversational grounding phenomena, such as lexical alignment, pronominal reference, and ellipsis, influence user trust and preference.
Methods
The researchers employed a multi-faceted approach to analyze conversational grounding phenomena and their impact on trust.
- Automatic Annotation: They automatically annotated linguistic properties of responses generated by various LLMs (GPT-3 variants) and retrieval-based models on the TopiOCQA dataset. This included identifying lexical alignment, syntactic form (short, fragment, long), and pronominal usage.
- Human Evaluation:
- Faithfulness Judgment: Annotators assessed the plausibility, groundedness, and faithfulness of system responses against dialogue history and knowledge sources.
- Preference Task: Annotators were presented with pairs of responses (often a baseline vs. a model variant) and asked to choose their preferred one, with access to the grounding document.
- Trust Study: Annotators were shown dialogue snippets and asked to choose the response they found most trustworthy, without direct access to the grounding document. This was done for various linguistic phenomena.
- Model Comparison: They compared open-book (retriever + reader) and closed-book (LLM only) models, including fine-tuned models and vanilla GPT-3 variants.
Datasets
- TopiOCQA: The development set of the TopiOCQA dataset was used for training and evaluation. This dataset comprises 205 information-seeking dialogues with 2514 turns, focusing on conversational QA with topic switching.
Key Insights and Ideas
- The "Stochastic Parrot" Paradox: LLMs that mimic human conversational patterns, like repeating user phrases (lexical alignment), are paradoxically preferred and trusted more, even when their responses are factually incorrect. This highlights a significant danger where superficial linguistic cues can override accuracy.
- Lexical Alignment is King (for Trust): High lexical alignment strongly correlates with increased user trust. Conversely, phenomena like ellipsis (fragments) and pronominal references are dis-preferred in terms of trust.
- GPT-3's Deceptive Fluency: Vanilla GPT-3 variants, while often more verbose and lexically aligned, are less faithful than fine-tuned models like DPR+FiD. This suggests that their conversational style, while appealing, can mask underlying inaccuracies.
- Human vs. System Language: Human responses in TopiOCQA tend to be more elliptical and use more pronominals, a style that fine-tuned models can emulate. However, LLMs often deviate from this, producing more sentential and lexically aligned outputs.
- Trust vs. Accuracy Trade-off: The preference task revealed that users often preferred unfaithful responses over faithful ones, or rejected both, indicating that conversational appeal can trump the need for accuracy.
Key Results
- Faithfulness: Automatic metrics and human judgments confirmed that GPT-3 variants were less faithful than the baseline DPR+FiD model.
- Preference: DPR+GPT-3 was preferred 70% of the time, with 48% of these preferences being for unfaithful responses. GPT-3 was preferred 45% of the time, with 66% of these being unfaithful.
- Trust: Responses with high lexical alignment were trusted significantly more (58% in the trust study) compared to those with low lexical alignment (32%). Short answers were also trusted more (66%) than fragments (26%).
- Linguistic Phenomena: Table 1 shows that GPT-3 variants produced more verbose and lexically aligned responses, while the fine-tuned model (DPR+FiD) generated shorter, fragmented responses with more pronominals, emulating human patterns.
Limitations and Future Work
- Limited Phenomena: The study focused on a subset of grounding phenomena (lexical alignment, ellipsis, pronouns), excluding others like repairs or clarification requests.
- Simplified Alignment: Lexical alignment was analyzed only over the last two turns of a conversation, potentially missing dynamic convergence over longer dialogues.
- Dataset Bias: The findings might be influenced by the specific domain and characteristics of the TopiOCQA dataset.
- Organic Use Cases: The crowd-sourced experiments, while controlled, might not fully reflect real-world, organic conversational scenarios.
- Future Work: The authors suggest further investigation into extrinsic evaluations to capture more organic use cases and exploring other grounding phenomena. They also note the need for future work on fact-checking extrinsic hallucinations. The ethical implications of LLMs being trusted due to superficial linguistic cues are also highlighted as a critical area.
Summary updated