Most of what we know about how models handle sarcasm comes from "laboratory" data. These are tweets labeled by a #sarcasm hashtag or by outside annotators, usually split 50/50 and read without the author's context. Real use looks different. In "field" data, the author says what they meant, sarcasm is rare, and the cue often sits outside the text. The gap between the two settings is large. Models that score F 0.874 on hashtag benchmarks fall to 0.364 on author-intended sarcasm (iSarcasm). Geotechnical engineering has a close parallel. Shear tests on small specimens remove the coarse rock skeleton, and so they miss a collapse in friction angle from 34.7° to 7.7° under saturation that only full-size in-situ boxes reveal (large-scale direct shear on soil-rock mixtures). The lesson there is not that the samples were too small. It is that screening for clean specimens removed the structure that governs behavior in the field.
We propose to ask the same question one level down, inside a language model. When interpretability tools find the neurons that make an LLM register "this is sarcastic," do those neurons belong to a general mechanism for semantic inversion (saying the opposite of what is meant)? Or are they lab-specific shortcuts tied to the hashtag labeling channel? Our measuring instrument is a multi-objective evolutionary search over sparse sets of neurons to switch off. It looks for sets that disrupt sarcasm judgments on lab data while leaving field data untouched, and the reverse, with damage to unrelated abilities held fixed. The trade-off curve this search traces between "lab effect" and "field effect" is a direct, quantitative picture of how much mechanism the two settings share.
Core Claim
Our hypothesis is that a substantial part of the internal machinery behind LLM sarcasm judgments on hashtag-style data is dissociable from the machinery used on author-intended sarcasm. If so, interpretability findings calibrated on lab corpora will not transfer, much as coreflood parameters fail at reservoir scale.
The test depends on minimally different pairs. Both settings provide a sarcastic text paired with a literal rewrite. The Sarcasm SIGN parallel corpus gives 3,000 #sarcasm tweets with literal translations written by outside annotators; this is the lab side. iSarcasm, as used in SemEval-2022 Task 6 Subtask C, gives tweets with rephrasings written by the authors themselves; this is the field side.
On each pair we measure the model's sarcasm "logit gap." The model is prompted to say whether a tweet is sarcastic, and the gap is the difference in its yes-versus-no score between the sarcastic text and its rewrite. Because this is a paired measure, it sidesteps the 1:5 class imbalance and the choice of decision threshold. It also controls much of the lexical content, so a neuron set that shrinks the gap is acting on the inversion itself rather than on topic words.
The new contribution is not a better sarcasm classifier. It is a measured map of which inversion components are shared across the lab-to-field boundary, and a demonstration that this map has a shape a single-score ranking would miss.
Research Program
Models. We work with LLaMA-3-8B and Qwen-2.5, the models dissected in the RAG knowledge-utilization study. We then repeat the core analysis on LLaMA3-70B to test whether conclusions themselves change with model scale.
Narrowing the search space. The first stage decides where in the layer stack to look. Geometry of Decision Making in Language Models found that intrinsic dimension rises and then falls across layers (a "hunchback" shape). Intrinsic dimension is the number of directions the hidden states effectively spread along. That paper found the peak marks where the model commits to an answer, but it studied only multiple-choice final tokens. We compute the same profile with its estimators (GRIDE and maximum likelihood) on the final-token states of sarcastic texts and their rewrites, separately for lab and field pairs. One secondary prediction is that field sarcasm shifts the peak later, meaning the model commits later when the cue is subtle. The main purpose of this stage, though, is practical: it fixes a window of MLP layers around the peak, which is where the RAG study located its conflict-resolving neurons.
Building the candidate pool. Within that window we score neurons by how differently they activate on sarcastic versus literal members of each pair. The scoring follows the entropy-of-activation idea behind KAPE in the RAG study. We keep a pool of a few thousand neurons drawn from both lab and field pairs.
The search. A candidate solution is a subset of this pool, and the intervention replaces those neurons' activations with their mean. Each subset receives three scores:
- the reduction in the paired logit gap on lab pairs;
- the reduction on field pairs;
- the accuracy lost on control tasks.
The control tasks are BoolQ and ARC-Easy, taken from the layer-importance study, plus literal-sentiment classification on the rewrites alone. That last task checks that we have not simply destroyed the model's sense of sentiment.
Control damage is imposed as a hard limit of one accuracy point, and subset size is capped. This keeps each search at two objectives and avoids the known failure of Pareto ranking when there are many objectives. We run three searches with NSGA-II, a standard evolutionary algorithm that keeps a population spread along the trade-off curve:
- one maximizing lab effect while minimizing field effect;
- the mirror image;
- one maximizing both, which finds the shared core.
From MEoH we borrow a selection rule that rewards both dominance and dissimilarity. MEoH enforced structural difference between programs using syntax-tree distance; here, Jaccard distance between neuron sets plays that role and keeps the population from collapsing onto one set. Each evaluation is a batch of forward passes over a fixed subsample of a few hundred pairs, which keeps the cost manageable. If the budget binds, we will add the Pareto-compliant surrogate.
Why a Pareto search rather than a weighted sum. The set of achievable (lab effect, field effect) pairs has no reason to be convex. Minimizing a weighted sum cannot reach compromise solutions that sit in concave pockets of the trade-off curve, as ERKM and two-objective rule selection both argue. Those pockets are exactly the partially shared mechanisms we want to find.
Baselines
Naive reference: random neuron sets. Random sets of matched size are drawn from the same layer window and mean-ablated in the same way. They establish how much any sparse ablation moves the logit gap. An evolved set counts as informative only if its effect lies outside the 95th percentile of the random distribution.
Strongest existing attribution approach: ranked-neuron sweep. Neurons are ranked by a single attribution score, either the KAPE-style contrast or the gradient of the logit gap. We then sweep the top-k, scoring "lab minus field" for the dissociation searches and "lab plus field" for the shared search. This is the strongest competitor because it is how neuron-level attribution is done in the RAG study, and it is far cheaper than evolution. It gets the same candidate pool, the same control constraint, and the same total budget of forward passes. We also add a weighted-sum variant, re-ranking under ten weightings, so that tuning is matched. Evolution beats this baseline only if its trade-off curve has clearly larger hypervolume (the area of objective space the curve dominates) on held-out pairs, with confidence intervals from resampling pairs.
Coarse attribution: layer-level Shapley ablation. We run the truncated-Shapley procedure of Investigating Layer Importance on lab and field pairs. That paper found that early "cornerstone" layers carry 29.1% of importance in LLaMA3-8B, and that ablating them costs 29.3% accuracy on average. Our prediction is that cornerstone layers will be the same for lab and field data and will fail the control constraint. That would show layer granularity is too coarse to see the dissociation.
Ablation of the key ingredient: paired measure versus unpaired. We rerun the best search using unpaired detection F1 on Ptacek versus iSarcasmEval in place of the paired logit gap. This tests whether controlling content through the rewrite pairs is what makes any dissociation genuine rather than topical.
Detection-level anchors. These place our readout model's sarcasm judgments in context. On iSarcasmEval Subtask A, the winning stce ensemble reports a sarcastic-class F1 of 0.6052 (macro F1 0.7675), against a BERT baseline at 0.348 (SemEval-2022 Task 6). We are not trying to beat 0.6052. We report the model's own F1 so readers know how competent the mechanism under study is. No reported result was found on Lacuna for neuron-level interventions on sarcasm, so our random-set and ranked-sweep results will be the first reference points.
Why not just…?
Why not just fine-tune a detector on iSarcasmEval and be done? Fine-tuning answers a different question. Even the best fine-tuned ensemble reaches only 0.6052 sarcastic-class F1 (stce), and its score says nothing about whether the representations it reuses were shaped by lab-style cues. Fine-tuning also does not tell interpretability researchers whether their lab-calibrated circuits are trustworthy, which is the point of this work.
Why not just train linear probes on each layer and check lab-to-field transfer? We do run this, as a cheap first test. It is a real threat to the more expensive parts of the project: if probes trained on lab pairs transfer perfectly to field pairs, the dissociation hypothesis is already in trouble. Probes do have a limitation, though. They detect where information can be read out, not which components the model actually uses. A probe can transfer while the causally used neurons differ. The intervention searches measure use, so we report both results and treat any disagreement between them as a finding in its own right.
Why not just rank neurons by attribution and ablate the top k? This is the most serious alternative, and it is built into the design as the main baseline. If a ranked sweep matches the evolved curve's hypervolume on held-out pairs, we will conclude that the trade-off region is effectively convex and that evolution is unnecessary. We would then report the dissociation using the cheaper method. Evolution has to earn its place empirically.
Why not just use layer Shapley values? The layer-importance results suggest these mostly pick up general-purpose early layers, whose removal reduces the model to guessing. Commitment, however, happens around the intrinsic-dimension peak (Geometry of Decision Making). The Shapley baseline will show whether layer-level attribution can separate lab from field at all.
Decisive Experiments
The central result is the dissociation search's trade-off curve on held-out pairs. The hypothesis is supported if neuron sets exist that remove at least half of the lab logit gap, change the field gap by less than a tenth, and stay within the one-point control limit. These sets must also beat both random and ranked baselines. If instead the "maximize both" search dominates and the dissociation curves hug the origin, mechanisms are shared, and lab-calibrated interpretability is vindicated for this task. That outcome is equally publishable.
Four further checks guard against artifacts:
- Content check. Lab-specific neurons should respond to features of the labeling channel and not to topic. We test this by measuring their effect on Riloff tweets, which are labeled by third parties but not by hashtag, and on iSarcasm items that outside annotators misperceived. iSarcasm reports that such annotators miss 30% of intended sarcasm.
- Scale check. We repeat everything on LLaMA3-70B to see whether the shared fraction grows with model size, as greater layer-importance concentration at larger scale might predict.
- Geometry check. We test whether the intrinsic-dimension peak for field pairs comes later than for lab pairs, linking the geometric picture to the causal one.
- Downstream check. We ablate the shared core and measure sarcastic-to-literal rewriting quality on the SIGN corpus. SIGN-context reaches 46.2% intended-sentiment accuracy against 25.7% for Moses (Sarcasm SIGN). This tests whether the neurons the model uses to detect inversion are also the ones it uses to undo it.
References
- iSarcasm: A Dataset of Intended Sarcasm
- Study on large scale direct shear test on soil-rock mixture with multiple grain grades in high fill slope engineering
- Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation
- SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic
- Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation
- Geometry of Decision Making in Language Models
- Investigating Layer Importance in Large Language Models
- Multi-Objective Evolution of Heuristic Using Large Language Model
- A Pareto-Compliant Surrogate Approach for Multiobjective Optimization
- Explainable recommendation based on knowledge graph and multi-objective optimization
- Selection of Linguistic Classfication Rules Using Two-objective Genetic Algorithms
- stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets