Whose Sarcasm Did the Model Memorize? Separating Perception Bias from Contamination

Sarcasm benchmarks come in two kinds. Some are labeled by the author, who says whether they meant a post sarcastically. Others are labeled by outsiders: crowd annotators, or the author's own #sarcasm hashtag stripped out and reused as a label. Models do far better on the second kind. The field reads this gap as proof that what an author means and what a stranger perceives are different things. This note argues that the gap has a second, unmeasured cause. For hashtag datasets, the label was written into the public tweet itself, so any model pretrained on scraped social media has seen each test item together with its answer. For author-labeled datasets such as iSarcasm, the tweet may be public but the author's private judgment normally is not.

We propose to separate these two causes. The first step is a fresh, never-published collection of posts written after the models' training cutoffs. Each post carries two labels: the author's own label and a majority label from third-party annotators. The second step is a set of controlled contamination experiments that expose open-weight models to test items with the label either visible in the text or hidden. Together these tell us how much of the reported "perception gap" is linguistic, and how much comes from the fact that hashtag labels leak into pretraining text while author-intent labels do not. They also give a direct measurement of whether modern language models side with the audience or with the author when the two disagree.

Core Claim

"Contamination" here means a model has seen test items, or their answers, during training. Contamination of pragmatic benchmarks is asymmetric in a way that has no analogue in the multiple-choice and code audits done so far. The contamination survey Benchmark Data Contamination of Large Language Models: A Survey distinguishes seeing test questions ("data level") from seeing questions with answers ("label level"). Distant-supervision sarcasm datasets are label-level contaminated by construction, because the hashtag is the answer and it sits in the scraped text. Author-intended datasets are at most data-level contaminated, unless their GitHub releases were ingested.

We therefore claim that the drop from 0.874 F1 on hashtag data to at most 0.364 on iSarcasm, reported by iSarcasm: A Dataset of Intended Sarcasm, overstates the true linguistic divergence. We also claim that the true divergence can only be measured cleanly by scoring the same model on the same posts against the author's label and the audience's label.

The measured quantity is a model's perception bias on a fresh twin-labeled set. Let $F_{\text{perc}}$ be the model's F1 on the sarcastic class against the third-party majority label, and $F_{\text{int}}$ its F1 against the author's label, both on identical items:

$$B = F_{\text{perc}} - F_{\text{int}}.$$

A positive $B$ means the model reads text the way a stranger does rather than the way its author meant it. Because $B$ is computed on uncontaminated items, it is a contamination-free estimate of the divergence. We can then compare it with the benchmark-level gap measured on old, possibly contaminated data.

Relation to Prior Work

The intent-perception literature motivates the question but does not examine exposure.

Four papers establish that intended and perceived sarcasm differ. The iSarcasm dataset and its SemEval successor, SemEval-2022 Task 6: iSarcasmEval, built author-labeled ground truth and showed that third-party annotators reach only 0.616 F1 against it. Exploring Author Context for Detecting Intended vs Perceived Sarcasm showed that author history alone predicts hashtag-style intended sarcasm at 0.922 F1 but helps little on perceived labels. Reactive Supervision found that first-person versus third-person reactions carry separable textual signatures. Perceived and Intended Sarcasm Detection with Graph Attention Networks found that user graphs help intent more than perception.

All four compare different datasets for the two notions: Ptacek versus Riloff, or SPIRS intended versus SPIRS perceived. Dataset source, time period and collection method are therefore confounded with label type, and none asks what pretrained models have already seen. Our twin-label design removes the dataset confound. The exposure experiments add the missing variable.

The 0.922 F1 without reading the target tweet is itself worth a second look. It shows that hashtag-style "intended" labels are highly predictable from who posted, and a pretrained model could have memorized exactly that for prolific users.

The contamination literature supplies our methods but has only audited factual, math and code benchmarks. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples showed that fine-tuning Llama-2-13B on rephrased MMLU test items raised its score from about 55 to about 86, and that n-gram filters miss such leakage. We reuse its controlled-injection design and its rephrasing condition. We add a manipulation that has no counterpart in multiple-choice benchmarks: whether the label is visible inside the exposed text.

Dynamic benchmarks attack contamination by regenerating tests, but not for subjective tasks. DyVal generates formal reasoning problems from graphs and states that the approach does not extend to pragmatic tasks. LiveBench refreshes questions over time.

The closest existing benchmarks work at a different level of analysis. PUB measures third-party pragmatic comprehension on static items and finds a 25% human–model gap on figurative language plus sensitivity to lexical hints. FANToM tracks who knows which facts in a conversation. Neither scores a model against an author's own intent, and neither audits exposure.

A constraint comes from Back to Square One, which showed that paired Winograd items can be solved from surface artifacts. We borrow its partial-input baselines to check the pairwise iSarcasmEval subtask.

Research Program

Exposure audit. The first component audits existing sarcasm benchmarks for exposure: iSarcasm, iSarcasmEval (English and Arabic), Ptacek, Riloff and SPIRS. We probe open-weight models whose training cutoffs postdate these releases in three ways.

The first probe tests verbatim recall: prompted with a tweet prefix, does the model complete the suffix? The second probes the label specific to hashtag data: shown a Ptacek tweet with its hashtag removed, does the model assign unusually high probability to #sarcasm as the continuation, compared with time-matched fresh tweets? The third tests format recall: prompted in the released file format, does the model reproduce the dataset's labels?

Perplexity is compared against length- and topic-matched fresh posts. This follows the matching- and comparison-based detectors catalogued in the contamination survey.

Twin-label collection. The second component is a fresh twin-label collection. We follow the recruitment protocol of iSarcasmEval: Prolific for English, and Appen for Arabic as an extension. Authors label posts they wrote after the latest cutoff among the evaluated models, and provide a literal rephrase and a short explanation as in iSarcasm. Each post is also labeled by at least five third-party annotators who see only the post, which gives both a majority perceived label and a disagreement rate.

We target about 3,000 English posts, comparable to iSarcasm's 4,484, and oversample sarcasm to soften iSarcasm's 1:5 class imbalance. The set is never released publicly as text. It is held behind gated evaluation access so that it stays uncontaminated.

Controlled contamination. The third component injects exposure under control. We continue pretraining open-weight models on a slice of the fresh set in four arms. In the first, text appears with no label. In the second, text appears with an inline hashtag-style label, which mimics distant supervision. In the third, text appears with its author label in a dataset-file format. The fourth is the third arm after rephrasing, following Yang et al.

Each arm is then scored against both the author and the perceived labels on the exposed items and on unexposed held-out items. Because everything else is fixed, this measures how much each kind of exposure inflates performance for each label type.

Decomposition. The fourth component combines the pieces. The benchmark-level gap (hashtag F1 minus iSarcasm F1) is compared with $B$ from the fresh set. The difference estimates the part attributable to exposure and dataset confounds. The injection arms tell us whether label-visible exposure alone can reproduce that difference in magnitude.

Baselines

The naive reference point is a classifier that labels every post sarcastic. It sets the floor for F1 on the sarcastic class under each label type. We also report the third-party human F1 of 0.616 against author labels from iSarcasm. On the fresh set we recompute that human figure directly from our own annotators, which serves as the human perception bias against which model $B$ is read.

The strongest existing system on intended sarcasm is the top iSarcasmEval English Subtask A entry, "stce", which reached an F1 on the sarcastic class of 0.605. The human majority vote reached 0.613, as reported in Sarcasm Detection is Way Too Easy!. No reported result was found on Lacuna for zero-shot or prompted large language models on iSarcasmEval or iSarcasm, so we establish those numbers ourselves.

To keep the comparison fair, we reproduce a fine-tuned encoder of the stce type, trained on the iSarcasmEval training split with the same tuning budget as our prompted-model runs. We score it on the iSarcasmEval test set and on our fresh set against both labels. On iSarcasmEval, "beating" is not our goal. What would count as a result is showing that prompted models lose more between old and fresh author-labeled data than this fine-tuned encoder does, because the encoder cannot have memorized the test set during pretraining in the same way.

The author-context baseline is the history-based cascade approach from Exploring Author Context for Detecting Intended vs Perceived Sarcasm, rebuilt for fresh authors. Where authors consent to sharing timelines, it tests whether history still predicts intent when the users cannot have been memorized.

The ablation that isolates our key ingredient is the contrast between the label-hidden and label-visible injection arms at identical token counts. If both arms inflate scores equally, label visibility is not the mechanism and the core claim fails.

Why not just…?

Why not just run a membership-inference or n-gram audit on iSarcasm and stop? Such an audit is our first component and is worth doing on its own, but it answers a narrower question. It says whether a model has seen items, not whether exposure changes the measured intent-perception gap. It also cannot be run against the training corpora of closed models. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples further shows that string matching misses paraphrased leakage. Only the fresh set and the injection arms establish effect sizes.

Why not just evaluate on posts from after the training cutoff? This is the real threat, since chronological comparison is the standard detector in the contamination survey. A plain before/after comparison mixes contamination with drift in topics, slang and platform culture. On its own it also reuses the dataset confound that plagues the sarcasm literature, because the old and new data differ in label source. Our design adds two controls. First, twin labels on the same fresh items measure perception bias with time held fixed. Second, the injection experiment exposes some fresh items and not others, so any score difference between exposed and unexposed items at the same time point is attributable to exposure rather than drift. If the time-only comparison gives the same decomposition as the full design, we will report that the cheaper method suffices.

Why not just generate synthetic sarcastic and literal posts with a language model, as a dynamic benchmark would? A generated post has no author whose private intent differs from its surface reading. Its "intended" label is the generator's own perception, which removes the phenomenon we are trying to measure. The survey of rephrased contamination found that 12.8% of CodeAlpaca was rephrased HumanEval, because generators reproduce what they memorized. Preference Leakage reports analogous leakage for language-model judges. We will test this directly: a model's $B$ on generator-written items should shrink toward zero relative to its $B$ on real authored items.

Why not just give models more conversational context? Context helps perception but does not reveal private mood or background, a point argued in Magnets for Sarcasm. Humans with full first-party pairs still reach only 0.613 F1. Context is also orthogonal to our question: we can supply the same thread context under both labels and see whether $B$ persists.

Proposed Experiments and Decisive Checks

We will evaluate several open-weight models, and closed models where accessible, in zero-shot and few-shot settings. Each is scored on the old benchmarks and on the fresh twin-labeled set, using F1 on the sarcastic class as in iSarcasmEval.

A secondary check applies partial-input baselines from Back to Square One to iSarcasmEval Subtask C, where top systems reach 0.870 accuracy in English. In this pairwise task a model must pick the sarcastic post over the author's literal rephrase. The partial-input variant scores each candidate alone, or uses only a sentiment-polarity difference between the two. If a polarity-only scorer approaches the published accuracy, Subtask C mostly measures a surface artifact rather than intent, and we will repeat the task on fresh pairs.

Three predictions decide the claim.

  1. Label-visible exposure. On the hashtag probe, models should assign elevated probability to the sarcasm hashtag on Ptacek tweets relative to fresh matched tweets, but show no comparable label recall on iSarcasm.
  2. Size of the fresh-set bias. $B$ on the fresh set should be positive for large models, meaning they side with strangers, but smaller than the old benchmark-level gap.
  3. Injection effect. The label-visible injection arm should inflate agreement with the injected label on exposed items well beyond the label-hidden arm.

If the probes show no hashtag recall and $B$ matches the old gap, contamination plays no role. The perception-gap literature then stands vindicated with a clean, contamination-free measurement, and that result is publishable in its own right. If instead $B$ is near zero while old benchmarks show a large gap, a substantial part of what the field has called intent-perception divergence is an artifact of which labels leaked onto the web.