Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers

Lorenzo Pacchiardi, Marko Tesic, Lucy G Cheke, Jose Hernandez-OralloSubmission to ICLR 2025 · Rejected

TL;DR

Many standard LLM benchmarks suffer from the "Clever Hans" effect: simple surface-level statistical cues, such as word frequencies and readability scores, can accurately predict ground-truth answers without any actual task reasoning. Furthermore, prominent LLM families (including models from OpenAI, Meta, and Mistral) perform significantly better on prompt instances that simple linear n-gram classifiers can solve, indicating that frontier models may often succeed via superficial shortcuts rather than genuine linguistic or logical competence.

What Problem Does This Paper Address?

In experimental psychology, the Clever Hans effect describes an animal appearing to perform arithmetic while actually responding to unconscious cues from the handler. A similar failure mode threatens artificial intelligence evaluation: if a benchmark contains spurious correlations between superficial prompt features and the target label, models can exploit these artifacts to achieve impressive accuracy without engaging the reasoning capabilities under evaluation.

While prior research demonstrated that individual keywords (such as the word "not") could game older NLP benchmarks, modern LLMs are far more expressive. This paper investigates whether combinations of shallow textual features—specifically unigrams, bigrams, and readability statistics—allow trivial linear models to game contemporary multiple-choice LLM benchmarks, and whether state-of-the-art LLMs are actively leaning on those same shortcuts.

Methods

The authors evaluate the internal validity of LLM benchmarks through a lightweight probing framework:

  1. Feature Extraction: For each prompt, the authors extract 13 distinct feature vectors:
  2. 12 $n$-gram representations varying across $n$-gram order (unigrams vs. unigrams + bigrams), tokenization level (word-level vs. subword GPT-2 token level), and feature weighting (Term Frequency, TF-IDF, or binary Presence).
  3. 1 text-metric vector containing readability and diversity indicators (e.g., Flesch Reading Ease, Gunning Fog Index, SMOG Index, Yule's $K$).
  4. Linear Classification: Regularized logistic regression classifiers ($L_1$ and $L_2$ penalties) are trained on prompt features to predict the correct label. Models are evaluated using Cohen's $\kappa$ to penalize chance agreement across binary and multiclass tasks.
  5. Behavioral Stratification of LLMs: The test prompts are partitioned into two subsets: instances successfully solved by the simple linear model vs. instances that failed. The authors test 44 LLMs from 11 model families across these splits using one-sided paired $t$-tests (with Benjamini–Hochberg False Discovery Rate correction) to determine whether LLMs show a systematic performance boost on shortcut-predictable instances.

Datasets

The study assesses 19 multiple-choice benchmarks where label options remain invariant across instances:

  • BIG-Bench tasks: Fantasy Reasoning, Moral Permissibility, Causal Judgment, Metaphor Boolean, bAbI Task 16, Formal Fallacies Syllogisms Negation.
  • LegalBench tasks: Abercrombie, Corporate Lobbying, Function of Decision Section, International Citizenship Questions, PROA.
  • Other reasoning/NLI benchmarks: CLadder, ProntoQA, ANLI, ART, Commonsense QA 2.0, NeuBAROCO, SpaceNLI, and WANLI.

No new benchmark was created; instead, the authors analyzed public datasets and evaluation logs from HELM and original benchmark releases, supplementing them with new instance-level model evaluations where needed.

Key Insights and Ideas

  • Widespread benchmark leakage via superficial cues: In nearly half of the benchmarks (9 out of 19), simple prompt n-grams or readability statistics yield Cohen's $\kappa > 0.2$, showing measurable predictive signal without any semantic task comprehension.
  • Extreme shortcuts in specialized domains: In benchmarks like Corporate Lobbying and SpaceNLI, linear classifiers reached $\kappa > 0.6$ (substantial agreement), demonstrating that task labels are heavily confounded by vocabulary choice.
  • TF-IDF actively hurts shortcut detection: Standard TF-IDF weighting reduced predictive power compared to raw frequency or binary presence because IDF discounts high-frequency words—precisely where spurious dataset-wide artifacts tend to live.
  • Frontier LLMs correlate with simple heuristics: LLMs from OpenAI, Meta, and Mistral show statistically significant performance advantages on prompt instances solvable by simple linear baselines, implying that LLM benchmark accuracy may partially reflect shortcut learning acquired during pretraining or supervised fine-tuning.

Key Results

  • Feature Predictability: Classifiers achieved $\kappa > 0.2$ on 9 of 19 datasets. On SpaceNLI and Corporate Lobbying, linear classifiers attained $\kappa$ values exceeding 0.60.
  • Model Reliance: Paired $t$-tests across the stratified splits revealed significant advantages on shortcut-predictable instances for several major model families after Benjamini–Hochberg correction:
  • Meta: Adjusted $p \approx 0.059$
  • OpenAI: Adjusted $p \approx 0.059$
  • Mistral AI: Adjusted $p \approx 0.089$
  • Writer: Adjusted $p \approx 0.089$
  • Anthropic and Aleph Alpha exhibited weaker trends ($p \approx 0.158$ and $p \approx 0.199$, respectively), while smaller sample sizes in families like Google, Cohere, and AI21 limited statistical power.

Limitations and Future Work

  • Correlation vs. Causation: The higher accuracy of LLMs on n-gram-predictable instances could partially indicate that these questions are inherently simpler, rather than proving direct reliance on spurious tokens. Controlled adversarial perturbations (e.g., token stripping or counterfactual edits) are required to establish strict causality.
  • Narrow Feature Space: The linear probing set only investigated $n$-grams ($n \le 2$) and high-level readability metrics. Models likely exploit more intricate, higher-order surface artifacts not captured here.
  • Benchmark Coverage: Evaluation was restricted to multiple-choice benchmarks with identical choice sets across questions, leaving open-ended generation benchmarks unexamined.
  • Future Directions: The authors recommend that benchmark creators incorporate simple linear probing into dataset construction pipelines to flag and eliminate unintended feature–label correlations prior to release.

Summary updated

Abstract

The integrity of AI benchmarks is fundamental to accurately assess the capabilities of AI systems. The internal validity of these benchmarks—i.e., making sure they are free from confounding factors—is crucial for ensuring that they are measuring what they are designed to measure. In this paper, we explore a key issue related to internal validity: the possibility that AI systems can solve benchmarks in unintended ways, bypassing the capability being tested. This phenomenon, widely known in human and animal experiments, is often referred to as the ‘Clever Hans’ effect, where tasks are solved using spurious cues, often involving much simpler processes than those putatively assessed. Previous research suggests that language models can exhibit this behaviour as well. In several older Natural Language Processing (NLP) benchmarks, individual $n$-grams like “not” have been found to be highly predictive of the correct labels, and supervised NLP models have been shown to exploit these patterns. In this work, we investigate the extent to which simple $n$-grams extracted from benchmark instances can be combined to predict labels in modern multiple-choice benchmarks designed for LLMs, and whether LLMs might be using such $n$-gram patterns to solve these benchmarks. We show how simple classifiers trained on these $n$-grams can achieve high scores on several benchmarks, despite lacking the capabilities being tested. Additionally, we provide evidence that some modern LLMs might be using these superficial patterns to solve benchmarks. This suggests that the internal validity of these benchmarks may be compromised and caution should be exercised when interpreting LLM performance results on them.
BibTeX
@misc{pacchiardi2024leaving,
  author = {Lorenzo Pacchiardi and Marko Tesic and Lucy G Cheke and Jose Hernandez-Orallo},
  title = {{Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers}},
  year = {2024},
  eprint = {2410.11672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.11672},
}
Figure