Code-writing language models have become much better at fixing programs, and the main reason is external feedback. The model writes code, a compiler or test suite runs it, the failures go back to the model, and it revises. Sarcasm detection on social media has no comparable loop. The best systems are large ensembles of fine-tuned transformers that cannot be inspected. On sarcasm that the author intended, as opposed to sarcasm marked with a hashtag, they reach an F-score on the sarcastic class of only about 0.6. They learn surface cues that disappear once labels come from the authors themselves. This note proposes giving sarcasm detection the missing ingredient: an objective test suite built from what authors actually meant. A language model would write and repeatedly debug a small, readable Python "patch program" that corrects a base classifier's sarcasm decisions, with failing tests fed back as if they were compiler errors.
The central idea is that author-intent data supplies something like unit tests. The iSarcasmEval collection pairs each sarcastic tweet with its author's own non-sarcastic rephrasing. Each pair is therefore a minimal contrast, since the intended meaning is held fixed and only the sarcastic form changes. A correct detector must rank the original above the rephrase. A detector that keys on hashtags, exclamation marks, or positive words will fail many of these pairs, and the failing pairs can be shown to the model as counterexamples. Our bet is that this external, intent-grounded oracle is what makes iterative refinement work on a subjective task. Without it, a model judging its own reasoning about sarcasm would simply reinforce its own misreadings.
Core Claim
Iterative code refinement has helped wherever an independent judge exists, but it has been confined to tasks where such a judge is obvious. Raw GPT-4 resolves 1.74% of SWE-bench issues and the multi-agent loop in MAGIS resolves 13.94%. MEIC fixes 78% of functional hardware bugs by checking each patch in a simulator and rolling back regressions. The survey Code to Think, Think to Code names subjective, pragmatic language as the open frontier for code-based reasoning, because there is no obvious way to execute and check a judgment like "this is sarcastic."
Do Users Write More Insecure Code with AI Assistants? warns about the risk of looping without such a judge. In that study, developers iterating with Codex produced more vulnerable code: 36% versus 7% on SQL injection. They also grew more confident in it.
We claim that contrastive tests built from authors' own rephrasings are a usable judge for sarcasm. We further claim that a refinement loop driven by these tests, with rollback, will produce patch programs that fix a strong base classifier's shortcut errors on author-intended sarcasm. We do not expect those gains to appear when the loop is driven by the model's self-critique or by hashtag-labeled data. The new element is not "an LLM looks at tweets." It is treating author-intent minimal pairs as an execution oracle, and showing which feedback signal makes refinement work on a subjective task.
Research Program
The system has four parts.
The first part is a base classifier: a single fine-tuned DeBERTa-v3-large model trained on the iSarcasmEval English training split. It outputs a sarcasm probability.
The second part is a small, fixed library of deterministic helper functions that patch programs may call:
- a structural tweet parser in the spirit of the Twitter Building Blocks of Improving Twitter Retrieval by Exploiting Structural Information, which separates hashtags, mentions, retweet markers, URLs, and message text;
- normalization of character flooding ("soooo");
- polarity lexicon lookups like those in SeNTU;
- the base classifier's probability on any substring;
- a sentence-level sentiment scorer.
Patch programs may not call a language model at inference time. This keeps them cheap, deterministic, and auditable.
The third part is the test suite, drawn only from training data. It contains three kinds of tests:
- labeled tweets, where the program's decision must match the author's label;
- author-rephrase pairs, where the program's score on the original must exceed its score on the rephrase;
- invariance tests, where removing URLs, normalizing elongated words, or deleting a trailing
#sarcasmmust not flip a decision whose label does not depend on that cue.
The fourth part is the refinement loop. A code-capable language model is shown the base classifier's errors and writes a patch program, which is a function that takes a tweet and the base probability and returns a corrected decision. The program is run on the test suite. Every failing test is returned with an execution trace showing which branch fired, which lexicon hits occurred, and what scores the helpers produced. This trace plays the role of a compiler traceback. The model then revises. Following MEIC, each candidate is scored on a separate validation slice of the test suite. Any revision that lowers that score is reverted to the best checkpoint. Following LearnAct, which grows and debugs a library of code actions from execution failures, accepted helper functions are kept for later iterations.
Two analyses accompany the system. The first tracks gains across iterations. CodeTransOcean reports that compiler-feedback self-debugging gains flatten after 2–3 rounds (DSR@0 48.57% to DSR@3 52.57%). We expect the same pattern and will cap iterations accordingly. The second analysis reads the final programs and catalogs the shortcut errors they correct. This qualitative output is something an ensemble cannot provide.
Baselines
The naive reference point is the base DeBERTa-v3-large classifier with no patch, plus the task organizers' reference systems. SemEval-2022 Task 6: iSarcasmEval reports $F_{sarcastic}$ of 0.275 for its SVM baseline and 0.348 for its BERT baseline on English Subtask A. Any patched system must clearly exceed its own unpatched base. Beating the organizer baselines alone proves little.
The strongest existing method is the Subtask A winner, stce. It ensembles 15 transformers (DeBERTa-v3-large, RoBERTa-large, XLM-RoBERTa-large) trained with contrastive losses, multi-sample dropout, and adversarial training, and it reports $F_{sarcastic}$ 0.6052 and macro F1 0.7675. We will compare against these reported numbers on the official test set. We will also run a matched comparison: patching a single model versus stce's single-model components where we can reproduce them, and patching an ensemble we rebuild ourselves, all trained on the same data. Beating stce means exceeding 0.6052 $F_{sarcastic}$ when we patch a comparable ensemble. Alternatively, it means closing at least half the gap between a single model and the ensemble while adding only a readable program and no extra trained parameters.
The strongest competitor on the specific question of whether pair feedback matters is contrastive fine-tuning. Here the same author-rephrase pairs from the training split are added to the base model's training as a margin ranking loss. This uses exactly the information our loop uses, but in gradient form. Both systems get the same pairs, the same validation slice for model selection, and a comparable tuning budget: equal numbers of hyperparameter trials, and an equal number of loop runs.
The ablations isolate each ingredient of the loop:
- labeled-tweet tests only, removing the pair tests;
- self-critique only, where the model reads its program and the failing examples' text but receives no execution results or traces (this is the model-close condition);
- no rollback;
- a single-shot patch written with no refinement.
For language-model prompting baselines on iSarcasmEval, we found no reported result on Lacuna. We will run zero-shot and chain-of-thought prompting with the same backbone ourselves.
Why not just…?
Why not just ensemble more fine-tuned transformers? stce shows that ensembling works on binary detection. The same competition's Subtask B, which asks for the type of irony, topped out at macro F 0.1630. iSarcasm shows that strong models trained on hashtag labels fall from 0.874 to at most 0.364 F on author-intended sarcasm. Scale alone does not remove shortcuts. Our patch layer sits on top of an ensemble rather than replacing it, so the test is whether the patch still adds something to the strongest ensemble we can build.
Why not just fine-tune on the rephrase pairs instead? This is the real threat. If contrastive fine-tuning matches the patched system, then the loop's only remaining value is interpretability, and we will say so. The matched baseline above settles the question. One reason to expect a difference is that the training split holds only a few thousand pairs. That is little gradient signal for a large model but plenty of counterexamples for writing targeted rules. Another reason is that execution traces show the model why an error occurred.
Why not just prompt a large language model, perhaps with self-refinement? Self-refinement without an external check is exactly the loop that Perry et al. found makes errors and confidence grow together. CodeTransOcean also reports that chain-of-thought can push models toward local, line-by-line reasoning that misses global structure. The self-critique ablation and the direct prompting baselines measure this risk directly.
Why not just hand-write rules? SeNTU's static rule layer reached 57.06% on SemEval-2015 but only 45.18% on Twitter 2014 Sarcasm, because literal lexicon rules cannot see a sentiment flip. Our patch programs are rules too. The differences are that they are generated and verified against intent-based tests, and that they condition on the base model's scores rather than on lexicons alone.
Proposed Experiments
The primary evaluation is iSarcasmEval English Subtask A, reporting $F_{sarcastic}$ and macro F1 on the official test set. Rephrases from the test split are never used in the loop.
Subtask C, which asks which member of a pair is sarcastic and was won by X-PuDu at 0.870 accuracy, serves as a direct check that pair-trained programs generalize to unseen pairs. We do this by scoring both members and choosing the higher.
Subtask B, with a top macro F of 0.1630, tests whether a program can be given category branches such as rhetorical question or understatement. Here the type labels serve as tests.
Transfer is checked in three settings:
- the original iSarcasm data, where third-party humans reach only 0.616 F;
- Twitter 2014 Sarcasm sentiment, where the program's decision flips the polarity from the base sentiment scorer, compared against SeNTU's 45.18% and evaluated on polarity accuracy rather than surface overlap, following the warning in Sarcasm SIGN;
- Arabic Subtask A, where the top team reports 0.563, which tests whether the helper library carries over to another language.
Three outcomes decide whether the core claim holds.
First, the full loop must beat both the labeled-tweet-only ablation and the self-critique ablation on Subtask A and Subtask C. If self-critique does as well, the claim about an external judge is false.
Second, the patched system must at least match contrastive fine-tuning on the same pairs. If it does not, the contribution reduces to interpretability.
Third, a shortcut probe. We rerun the loop with hashtag-labeled training tests in place of author tests. We predict that the resulting programs will lose much of their gain on author-intended test data. That would mirror the 0.874-to-0.364 collapse and show that the source of the oracle, not the loop itself, carries the effect.
We will report results per iteration, expecting gains to flatten within about three rounds. We will report the rollback rate. We will also have two annotators read a sample of the final programs and estimate what fraction of the fired branches correspond to recognizable sarcasm cues rather than dataset artifacts.
References
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- MEIC: Re-thinking RTL Debug Automation using LLMs
- Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs
- Do Users Write More Insecure Code with AI Assistants?
- Improving Twitter Retrieval by Exploiting Structural Information
- SeNTU: Sentiment Analysis of Tweets by Combining a Rule-based Classifier with Supervised Learning
- Empowering Large Language Model Agents through Action Learning
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic
- stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets
- iSarcasm: A Dataset of Intended Sarcasm
- Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation