TL;DR
Many standard LLM benchmarks suffer from the "Clever Hans" effect: simple surface-level statistical cues, such as word frequencies and readability scores, can accurately predict ground-truth answers without any actual task reasoning. Furthermore, prominent LLM families (including models from OpenAI, Meta, and Mistral) perform significantly better on prompt instances that simple linear n-gram classifiers can solve, indicating that frontier models may often succeed via superficial shortcuts rather than genuine linguistic or logical competence.
What Problem Does This Paper Address?
In experimental psychology, the Clever Hans effect describes an animal appearing to perform arithmetic while actually responding to unconscious cues from the handler. A similar failure mode threatens artificial intelligence evaluation: if a benchmark contains spurious correlations between superficial prompt features and the target label, models can exploit these artifacts to achieve impressive accuracy without engaging the reasoning capabilities under evaluation.
While prior research demonstrated that individual keywords (such as the word "not") could game older NLP benchmarks, modern LLMs are far more expressive. This paper investigates whether combinations of shallow textual features—specifically unigrams, bigrams, and readability statistics—allow trivial linear models to game contemporary multiple-choice LLM benchmarks, and whether state-of-the-art LLMs are actively leaning on those same shortcuts.
Methods
The authors evaluate the internal validity of LLM benchmarks through a lightweight probing framework:
- Feature Extraction: For each prompt, the authors extract 13 distinct feature vectors:
- 12 $n$-gram representations varying across $n$-gram order (unigrams vs. unigrams + bigrams), tokenization level (word-level vs. subword GPT-2 token level), and feature weighting (Term Frequency, TF-IDF, or binary Presence).
- 1 text-metric vector containing readability and diversity indicators (e.g., Flesch Reading Ease, Gunning Fog Index, SMOG Index, Yule's $K$).
- Linear Classification: Regularized logistic regression classifiers ($L_1$ and $L_2$ penalties) are trained on prompt features to predict the correct label. Models are evaluated using Cohen's $\kappa$ to penalize chance agreement across binary and multiclass tasks.
- Behavioral Stratification of LLMs: The test prompts are partitioned into two subsets: instances successfully solved by the simple linear model vs. instances that failed. The authors test 44 LLMs from 11 model families across these splits using one-sided paired $t$-tests (with Benjamini–Hochberg False Discovery Rate correction) to determine whether LLMs show a systematic performance boost on shortcut-predictable instances.
Datasets
The study assesses 19 multiple-choice benchmarks where label options remain invariant across instances:
- BIG-Bench tasks: Fantasy Reasoning, Moral Permissibility, Causal Judgment, Metaphor Boolean, bAbI Task 16, Formal Fallacies Syllogisms Negation.
- LegalBench tasks: Abercrombie, Corporate Lobbying, Function of Decision Section, International Citizenship Questions, PROA.
- Other reasoning/NLI benchmarks: CLadder, ProntoQA, ANLI, ART, Commonsense QA 2.0, NeuBAROCO, SpaceNLI, and WANLI.
No new benchmark was created; instead, the authors analyzed public datasets and evaluation logs from HELM and original benchmark releases, supplementing them with new instance-level model evaluations where needed.
Key Insights and Ideas
- Widespread benchmark leakage via superficial cues: In nearly half of the benchmarks (9 out of 19), simple prompt n-grams or readability statistics yield Cohen's $\kappa > 0.2$, showing measurable predictive signal without any semantic task comprehension.
- Extreme shortcuts in specialized domains: In benchmarks like Corporate Lobbying and SpaceNLI, linear classifiers reached $\kappa > 0.6$ (substantial agreement), demonstrating that task labels are heavily confounded by vocabulary choice.
- TF-IDF actively hurts shortcut detection: Standard TF-IDF weighting reduced predictive power compared to raw frequency or binary presence because IDF discounts high-frequency words—precisely where spurious dataset-wide artifacts tend to live.
- Frontier LLMs correlate with simple heuristics: LLMs from OpenAI, Meta, and Mistral show statistically significant performance advantages on prompt instances solvable by simple linear baselines, implying that LLM benchmark accuracy may partially reflect shortcut learning acquired during pretraining or supervised fine-tuning.
Key Results
- Feature Predictability: Classifiers achieved $\kappa > 0.2$ on 9 of 19 datasets. On SpaceNLI and Corporate Lobbying, linear classifiers attained $\kappa$ values exceeding 0.60.
- Model Reliance: Paired $t$-tests across the stratified splits revealed significant advantages on shortcut-predictable instances for several major model families after Benjamini–Hochberg correction:
- Meta: Adjusted $p \approx 0.059$
- OpenAI: Adjusted $p \approx 0.059$
- Mistral AI: Adjusted $p \approx 0.089$
- Writer: Adjusted $p \approx 0.089$
- Anthropic and Aleph Alpha exhibited weaker trends ($p \approx 0.158$ and $p \approx 0.199$, respectively), while smaller sample sizes in families like Google, Cohere, and AI21 limited statistical power.
Limitations and Future Work
- Correlation vs. Causation: The higher accuracy of LLMs on n-gram-predictable instances could partially indicate that these questions are inherently simpler, rather than proving direct reliance on spurious tokens. Controlled adversarial perturbations (e.g., token stripping or counterfactual edits) are required to establish strict causality.
- Narrow Feature Space: The linear probing set only investigated $n$-grams ($n \le 2$) and high-level readability metrics. Models likely exploit more intricate, higher-order surface artifacts not captured here.
- Benchmark Coverage: Evaluation was restricted to multiple-choice benchmarks with identical choice sets across questions, leaving open-ended generation benchmarks unexamined.
- Future Directions: The authors recommend that benchmark creators incorporate simple linear probing into dataset construction pipelines to flag and eliminate unintended feature–label correlations prior to release.
Summary updated