TL;DR
This paper investigates whether Large Language Models (LLMs) truly possess "Theory of Mind" (ToM) capabilities or if their apparent understanding is a result of superficial heuristics, often termed the "Clever Hans Effect." By systematically testing LLMs on multiple-choice question (MCQ) benchmarks with adversarial reordering of answer choices, the authors demonstrate a significant susceptibility to positional bias, suggesting that reasoning abilities are not as robust as performance might initially suggest.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable abilities in understanding human intentions, emotions, and preferences, leading to the hypothesis that they possess a "Neural Theory of Mind" (N-TOM). However, previous research has raised concerns that these impressive performances might be due to superficial heuristics or "order bias" in the way questions and answers are presented, rather than genuine reasoning. This phenomenon, known as the "Clever Hans Effect," means that LLMs might be "fooled" by the presentation of information, leading to an overestimation of their true capabilities. The paper aims to rigorously test this hypothesis by systematically analyzing LLMs' resilience to such biases.
Methods
The authors employ a systematic evaluation approach using multiple-choice question (MCQ) benchmarks. Their core methodology involves creating adversarial prompts by altering the order of answer choices. Specifically, they test three prompt configurations:
- Original Order: The standard presentation of choices.
- First Choice: The correct answer is deliberately placed as the first option.
- Last Choice: The correct answer is deliberately placed as the last option.
They also explore the impact of few-shot prompting (providing examples in the prompt) and Chain-of-Thought (CoT) prompting (encouraging step-by-step reasoning) to see if these techniques can mitigate the observed biases. The evaluation focuses on measuring accuracy under these different conditions.
Datasets
The paper utilizes four established MCQ benchmarks:
- PIQA (Physical Interaction Question Answering): Focuses on everyday situations with binary choices.
- OpenBookQA: Requires multi-step reasoning, common knowledge, and text comprehension.
- CommonsenseQA: Deals with general commonsense knowledge.
- Social IQa: Assesses reasoning about social interactions and implications.
These datasets were chosen for their MCQ format, which facilitates prompt manipulation.
Key Insights and Ideas
- LLMs are highly susceptible to positional bias: The order of answer choices significantly impacts LLM performance, revealing a "Clever Hans Effect" rather than robust reasoning.
- Superficial heuristics over genuine reasoning: The observed performance gaps suggest that LLMs often rely on structural cues (like the position of an answer) rather than deep understanding.
- Few-shot prompting offers marginal improvement: While providing examples can slightly boost robustness, it doesn't fully eliminate the order bias and might even introduce new vulnerabilities.
- Chain-of-Thought (CoT) is a promising mitigation strategy: Encouraging step-by-step reasoning significantly stabilizes LLM performance and reduces the impact of positional bias, making models more robust.
- N-TOM abilities are not as robust as they appear: The findings challenge the notion of inherent N-TOM capabilities, suggesting that current LLMs might be mimicking understanding through learned patterns and biases.
Key Results
The study found significant performance drops when the target answer was not in the first position. For instance, in the zero-shot scenario, accuracy varied drastically based on choice order. For example, Alpaca's accuracy on OpenBookQA dropped from 36.2% (Original) to -9.2% when the target was the last choice. While few-shot prompting showed some improvement, CoT prompting demonstrated the most substantial gains in stability and robustness across models like Llama2 and Vicuna. The paper highlights that models like Alpaca and Falcon showed less improvement with CoT compared to Llama2 and Vicuna.
Limitations and Future Work
The authors acknowledge that their study focused on smaller, countable LLMs and that more extensive and robust models were not tested. They plan to:
- Extend their experimentation to larger models like GPT-3.5 and GPT-4.
- Investigate the impact of backbone model parameter variations and refinement techniques.
- Explore prompt engineering techniques like Chain-of-Thought and Tree-of-Thought in different contexts.
- Analyze whether the observed phenomena are due to structural representations or merely emergent features of LLMs.
- Develop new benchmarks that are less susceptible to these biases to better evaluate true abilities.
Summary updated