When a language model answers that an elephant is heavier than a fox, there are two possibilities. It may hold something like a model of size and weight. Or it may have read "big as an elephant" a million times and be completing a familiar phrase. Static commonsense benchmarks cannot tell these apart, and once they are published they are crawled into the next model's training data. This proposal uses similes, the "as X as a Y" expressions people use to name the extreme of a property, in two roles. Mined at scale, similes give an ordering of everyday objects by properties such as size, speed, hardness, or bravery, and from that ordering fresh comparison questions can be generated on demand. The corpus frequency of each simile also measures how strongly the model was exposed to the cliché behind each answer. Together these yield a commonsense test that can be regenerated whenever it leaks, and that reports how much of a model's score depends on recalling tropes.
The central device is a matched triplet of questions. The first is a stereotype-consistent comparison ("which is slower, a snail or a horse?"). The second has the same answer but its supporting simile is rare. The third is a counter-stereotype version, in which a short context reverses the answer ("the snail was taped to a racing drone; the horse was asleep in its stall"). A model that reasons about the property should handle all three. A model that recalls tropes will do well on the first, worse on the second, and fail the third, and the size of that failure should grow with how common the simile is.
Core Claim
Simile-mined intensity spectra support a procedurally generated benchmark for qualitative comparison, a domain where no contamination-free generator exists. The generated items also have a property that formal dynamic benchmarks lack: each item comes with a graded measure of its own "trope exposure." Our claim is falsifiable on two fronts. First, frontier models will show a dose-response pattern, with counter-stereotype accuracy falling as the log-frequency of the overridden simile rises. That pattern is the signature of recall rather than inference. Second, fine-tuning a model on rephrased copies of a released item set will inflate its score on that set but not on items regenerated from held-out concept pairs. This mirrors, for commonsense, the inflation that Rethinking Benchmark and Contamination for Language Models with Rephrased Samples showed on MMLU.
How This Relates to Prior Work
Two lines of work motivate the gap. On the contamination side, DyVal generates reasoning problems from graphs so that test items never exist before evaluation. It found that models strong on static tests score nearly zero on its tasks, but its grammar covers only arithmetic, logic, and graph search, and its authors name qualitative commonsense as unaddressed. A²RBench shares the formal-only restriction. We borrow their principle of generating at evaluation time and extend it to a domain where gold answers come from human consensus rather than a solver. The survey Benchmark Data Contamination of Large Language Models: A Survey separates contamination by level, from overlapping topics up to leaked labels. Trope recall sits at its weakest, semantic level, where no detector applies. That is why we measure exposure instead of trying to detect it.
On the commonsense side, Back to Square One showed that Winograd-style scores reflect artifacts. RoBERTa-large exceeds 60% with the entity candidates removed, and the top model falls from 89.71% to 79.41% when both twin sentences must be solved. We reuse its group-scoring idea: credit requires solving the whole triplet. Unlike that paper, we generate new items rather than re-scoring static ones. WinoGrande removes predictable items with adversarial filtering, and its authors concede that this can discard real commonsense, because a stereotype and a shortcut look alike to a classifier. COM2SENSE pairs true and false sentences and reports human pairwise accuracy of about 95% against about 51% for UnifiedQA-3B, but its perturbations are hand-crafted and its set is static. Our design resolves the tension WinoGrande leaves open. We keep stereotypes, because knowing that snails are slow is legitimate knowledge, and test only whether the model can override them when the context requires it.
The simile literature supplies the method. Ordering Concepts Based on Common Attribute Intensity combines simile, comparative, dependency, and co-occurrence signals in a Ranking SVM that orders concepts by attribute intensity. It reports average Spearman ρ = 0.441 against a PMI baseline of 0.274, with human consensus at ρ = 0.75. That paper is on Japanese blogs, and it notes that intensity judgments drift over time. MAPS-KB provides 4.3 million English topic–property–vehicle triplets, with plausibility and typicality scores, mined from OpenWebText, Gutenberg, and BookCorpus. We use it both as the source of vehicles and as the frequency source for trope exposure. Its implicit-property expansion relies on COMET and RoBERTa, so we restrict ourselves to its explicit triplets. Probing Simile Knowledge from Pre-trained Language Models showed that BERT's simile predictions are dominated by frequent words ("good" makes up 72.37% of its top-15 predictions). It did not ask whether correct simile predictions are memorized tropes; our dose-response analysis asks exactly that. Snoopy and Studying and improving reasoning in humans and machines tie few-shot accuracy to pretraining term frequency and low surprise. We sharpen that idea by indexing a specific phrase, the simile that licenses the answer, rather than individual terms.
Research Program
The first component is a spectrum builder. For each of roughly 60 English properties drawn from MAPS-KB, we collect candidate concepts and order them. Following Iwanari et al., we combine four signals: simile anchors from MAPS-KB, comparative patterns ("X is faster than Y") mined from the same open corpora, dependency-linked predications, and co-occurrence. We then fit a pairwise ranker. Before trusting the English port, we check it on Iwanari's own 35 Japanese concept sets, where the ρ = 0.441 figure is the target to match. For English, human validation is mandatory: five annotators order a sample of 40 spectra, and only concept pairs whose order exceeds 80% annotator agreement become gold. This avoids generating questions from the ranker's mistakes.
The second component is a generator. It samples a property and a gold-ordered concept pair, then renders the three triplet variants through a set of surface templates. For the counter-stereotype variant, a context sentence reverses the answer using an explicit, checkable device: a change of state, a scale, or an external aid. Humans verify that the context is decisive, and we expect to discard about a quarter of contexts. Each item is tagged with the MAPS-KB log-frequency of the simile that supports the stereotypical answer.
The third component is the analysis. For each model, we regress counter-stereotype accuracy on simile log-frequency and treat a negative slope as evidence of trope reliance. We use logistic regression because accuracy is binary per item, controlling for concept frequency so the effect is not just Snoopy-style term familiarity. We also report the gap between stereotype-consistent and counter-stereotype accuracy, and triplet-level group accuracy.
Baselines
The naive reference point is a PMI ranker. It answers every comparison by which concept co-occurs more with the property word, and it is the PMI baseline of Iwanari et al. (ρ = 0.274 on their data). It should do well on stereotype-consistent items and near zero on counter-stereotype items by construction. Any model that matches its profile is behaving like a co-occurrence table. We run it on exactly the generated items, with frequencies from the same corpora.
For the spectrum-construction step, the strongest existing method is the Ranking SVM of Ordering Concepts Based on Common Attribute Intensity, which reports average ρ = 0.441. Our port counts as faithful if it reaches a comparable correlation on their 35 sets. It counts as useful in English if annotator-validated spectra correlate with it at a level approaching the reported 0.75 human ceiling. No English number has been reported, so we report ours alongside annotator agreement.
For the benchmark itself, the strongest competing protocol is having a strong LLM write fresh comparison questions on request. This is how many dynamic commonsense sets are made in practice, and it is the cheapest way to get "unseen" items. We give it the same property list, item count, and a matched human-verification budget, and ask it to produce stereotype-consistent and counter-stereotype items. No reported result was found on Lacuna for an LLM-authored qualitative-comparison benchmark of this kind. The static references are WinoGrande and COM2SENSE pairwise accuracy, using the cited human figures of 94.0% and about 95%.
Two ablations isolate the key ingredient. The first replaces simile-based exposure tags with plain concept co-occurrence counts. If the dose-response slope survives unchanged, similes add nothing beyond term frequency. The second removes the counter-stereotype variant, which reduces the benchmark to a fresh but trope-friendly comparison set. We expect this version to overstate model competence. Our approach beats the LLM-authoring baseline if its items show lower near-duplicate rates against public commonsense sets and more spread in exposure, and if models show a measurable trope-reliance slope on our items that the LLM-written items fail to reveal.
Why not just…?
Why not just ask GPT-4 to write new commonsense questions? This is the real threat. Yang et al. found that 12.8% of the GPT-4-generated CodeAlpaca contains rephrased HumanEval problems, so teacher models reproduce what they memorized. Chen et al. show that language models drift toward frequent, clichéd completions. An LLM author will therefore tend to write items centered on the very tropes we want to control, and it gives no per-item exposure signal. We still run it as the main baseline. We measure its overlap with WinoGrande, COM2SENSE, and the LLM Decontaminator's embedding-plus-judge check, and we compare the distribution of simile frequency in its items against ours. If its items turn out equally diverse and equally revealing, then the simile pipeline is only a source of exposure tags, and the proposal shrinks to that.
Why not just evaluate on post-cutoff data, as LiveBench does? LiveBench and the chronological analyses in the survey rely on content that did not exist before the cutoff. Snails were slow before every cutoff, so qualitative commonsense has no post-cutoff version. Novelty here has to come from composition, meaning new pairings and new reversing contexts, which is what the generator supplies.
Why not just use co-occurrence or PMI and skip the similes? Iwanari et al. found PMI far weaker for intensity ordering (0.274 vs 0.441), and similes are what anchor the extremes of each spectrum. The first ablation tests whether simile frequency predicts trope reliance beyond co-occurrence. If it does not, we will report that similes matter only for building the spectra.
Why not just paraphrase existing benchmarks and n-gram decontaminate? Yang et al. show that n-gram filters miss rephrased leaks and that training on paraphrases lifted Llama-2-13B on MMLU from about 55 to about 86. Paraphrasing a static set leaves its answers and concept pairs unchanged, so the leaked knowledge still transfers.
Proposed Experiments
The decisive contamination check reproduces Yang et al.'s protocol in our domain. We release a frozen set of 2,000 triplets, fine-tune Llama-2-13B on rephrased copies of it, and compare score gains on three item sets: the frozen set, items regenerated from the same concept pairs, and items regenerated from held-out concept pairs. We predict large inflation on the frozen set, partial inflation on same-pair regenerations, and little on held-out pairs. Gains on held-out pairs would count as genuinely learned property knowledge rather than contamination. If inflation carries over fully to held-out pairs, the regeneration claim fails.
The decisive memorization check is the dose-response regression across open and API models. A significantly negative slope of counter-stereotype accuracy on simile log-frequency, after controlling for concept frequency, supports trope recall. A flat slope with high counter-stereotype accuracy would mean models reason about these properties, which is also worth reporting. As a secondary check, models fine-tuned on WinoGrande should improve on stereotype-consistent items but not on counter-stereotype items, extending Elazar et al.'s artifact finding to a generated setting. All gold labels on the reported items come from human verification, and annotator agreement is reported next to every model score.
References
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- Benchmark Data Contamination of Large Language Models: A Survey
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences
- Ordering Concepts Based on Common Attribute Intensity
- MAPS-KB: A Million-Scale Probabilistic Simile Knowledge Base
- Probing Simile Knowledge from Pre-trained Language Models
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- Studying and improving reasoning in humans and machines
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark