Synthetic Contexts for Real Replies in Low-Resource Sarcasm Detection

Sarcasm is a statement whose literal meaning is the opposite of what the speaker intends. Whether a reply is sarcastic usually depends on what came before it. "Great, just what I needed" is sincere after good news and sarcastic after a flat tire. Systems for Arabic, Bangla-English and other low-resource varieties are trained almost entirely on isolated posts, with very few sarcastic examples and no conversation history. They fail badly. A Bangla-English classifier reaches a sarcasm F1 of only 0.39 (MixSarc), and the best Arabic system in the iSarcasmEval shared task reaches an F-score on the sarcastic class of 0.563 (SemEval-2022 Task 6: iSarcasmEval). The usual remedy is to generate more sarcastic text. In a low-resource dialect, generated text is exactly what tends to come out unnatural and culturally off.

We propose the reverse. Keep every reply human-written, and generate only the preceding conversational turn. A real, sincere reply from the abundant majority class becomes a new sarcastic example by writing a prior turn under which it would be ironic. A real sarcastic reply gets a matched sincere context under which the same words are meant literally. Each pair shares an identical reply and carries opposite labels, so the only way to separate them is to read the context. This addresses the shortage of sarcastic examples and the missing context at once. Generated contexts are kept only if a context-aware rewriter, given the context and the reply, recovers the meaning the reply's author actually intended. The rewriter is a model that turns an utterance that depends on earlier turns into a self-contained statement, borrowed from conversational search.

Core Claim

The claim has two parts. First, for a low-resource language, augmentation that keeps the reply authentic and synthesizes only the conversational context will improve sarcastic-class F1 at the real-world sarcasm rate more than four alternatives:

  • token-level augmentation,
  • class reweighting,
  • prompting a large language model zero-shot,
  • fully generated sarcastic dialogues.

Second, the gain will come specifically from the paired, label-flipping structure. Adding the same number of unpaired synthetic contexts should help less. We think this is the right lever because it puts generation where generation errors matter least. A slightly stilted setup turn costs little; a stilted sarcastic punchline teaches the model the wrong cue.

Where this sits in prior work

Three papers come closest, and each leaves out a different piece.

Rhetorical Device-Aware Sarcasm Detection with Counterfactual Data Augmentation is closest in spirit. It uses GPT-4o to rewrite sarcastic sitcom dialogues into non-sarcastic minimal pairs and back, and reports an average 17% Macro F1 gain. It rewrites the sarcastic line itself, works only on English scripted dialogue, and evaluates on curated pairs rather than at the natural sarcasm rate. We hold the line fixed and rewrite the context; we work in low-resource varieties; and we evaluate at the natural sarcasm rate.

Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context retrieves labeled responses for unlabeled English conversation contexts and transfers their labels. On the FigLang2020 Twitter validation set this raised F1 from 0.854 to 0.897. It presupposes about 100,000 crawled real contexts, which low-resource varieties do not have; our method manufactures the contexts instead.

akaBERT balances the Arabic iSarcasmEval data by substituting words using AraBERT embeddings. It reports ensemble F1 rising from 77% to 93%. That figure appears to be an overall F1, not directly comparable to the official sarcastic-class F-score of 0.563, so we will re-run it under the official metric. Its augmentation never leaves the single tweet and never touches discourse.

Two lines of work motivate the design.

  • Context matters. Conditioning on prior turns supplies the incongruity signal (Hunter Gregory et al.). Three prior turns work better than the full history (Neural Sarcasm Detection using Conversation Context). Yet every low-resource sarcasm benchmark in our file is context-free, which leaves open whether the 0.39 collapse on MixSarc is a vocabulary problem or a missing-context problem.
  • Balanced benchmarks mislead. Abercrombie and Hovy show that F1 collapses when moving from balanced to naturally skewed (1:4) conversations while AUC barely moves. Khodak et al. document precision drops at natural distributions. Any augmentation that silently changes the training class ratio must therefore be corrected and judged at deployment prevalence.

Three results constrain the design.

  • Synthetic data must look like the target language. Augmentation helps only when synthetic samples match the target distribution (Ray Groshan et al.). This is the reason the reply stays authentic.
  • Sentiment features fail under imbalance. Feeding sentiment-clash scores into a classifier hurt performance on imbalanced iSarcasmEval (connotation_clashers). So we do not hand the classifier a sentiment feature; the context is plain text.
  • Author intent differs from annotator perception. iSarcasm shows third-party labels diverge from authorial intent. We therefore anchor filtering on author-written rephrasings wherever they exist.

Two pieces of method are borrowed.

  • Context-dependent rewriting. Pointer-generator rewriting (Rastogi et al.) and preference-trained rewriting (DualReform) turn a context-dependent utterance into a standalone one. These systems assume literal, cooperative, information-seeking turns. We repurpose the idea as a verifier whose target is the speaker's intended meaning, much as Sarcasm SIGN translates sarcasm into literal paraphrase, but conditioned on context.
  • Generating native-like text. Ivory Yang et al. show that language models can be prompted to produce native-like low-resource text, which we need only for the shorter, lower-stakes context turn.

Research Program

The generator. A multilingual instruction-tuned language model is prompted with a real reply and a target label. It writes a one-to-three-turn preceding context in the same language variety and in the same code-mixing style. For a sincere reply with a sarcastic target label, the prompt asks for a situation in which the reply's literal meaning is contradicted. For a sarcastic reply with a sincere target label, it asks for a situation in which the literal meaning is true. The reply may be minimally edited only to introduce a pronoun or ellipsis pointing into the new context. That edit is capped, by edit distance, and logged, so the reply stays essentially authentic. The three-turn ceiling follows the context-window finding cited above.

The verifier. A rewriter from a different model family than the generator receives the context and the reply. It outputs a standalone statement of what the speaker means. For iSarcasmEval sarcastic replies, the target meaning is the author's own non-sarcastic rephrase. For sincere replies the target is the reply itself, with references resolved. A sarcastic-target context is accepted only if a multilingual entailment model judges the rewrite to contradict the reply's literal reading. A sincere-target context is accepted only if the rewrite is entailed by it. The same check must flip across the two members of a pair.

We will audit a random sample of accepted and rejected items with native speakers. This measures how often the filter is right, rather than assuming it.

Training and the class ratio. The detector is a language-appropriate encoder (AraBERT or MARBERT for Arabic, Banglish-BERT for MixSarc) that reads the context and reply jointly. Because the augmentation changes the training class ratio, we correct for this at inference by logit adjustment, which shifts each class score by the log of how much its share changed:

$$\tilde{s}_y(x) = s_y(x) - \log \pi^{train}_y + \log \pi^{deploy}_y,$$

Here $s_y(x)$ is the model's score for class $y$, $\pi^{train}_y$ is that class's share of the augmented training set, and $\pi^{deploy}_y$ is its share at deployment, estimated from the natural development split. We also report a variant that tunes the decision threshold on a natural-ratio development split. Separating the two shows that any gain is not just a calibration effect.

Baselines

The naive reference is the same encoder fine-tuned on the original imbalanced data with plain cross-entropy and a threshold of 0.5. Published numbers anchor what this looks like: the official iSarcasmEval BERT baseline reaches an English sarcastic-class F-score of 0.348, and the MixSarc Banglish-BERT model reaches a sarcasm F1 of 0.3938.

The imbalance baseline is the same encoder with inverse-frequency class weights, focal loss, and threshold tuning on the natural development split. This is the cheapest serious competitor. Bashmal et al. and Sharma et al. argue reweighting cannot create the missing positive examples, but we will test that claim rather than assume it.

The strongest published low-resource method is the akaBERT embedding-substitution augmentation, reimplemented with NLPAug at the same augmented-set size as ours and scored under the official sarcastic-class F-score. On Arabic iSarcasmEval, the competition-best sarcastic-class F-score of 0.563 (top team CS-UM6P) is the external bar. We count our method as beating it only if we exceed 0.563 on the official Arabic test set without ensembling, or if, with ensembling, we beat a fair reimplementation of their pipeline.

The strongest context-generation competitor is a reimplementation of the Hong et al. duplex counterfactual augmentation. The same language model rewrites the sarcastic line itself, rather than the context, to produce pairs of the same count and target ratio. This baseline isolates the claim that keeping the reply authentic matters.

The zero-shot large-model reference is the same generator model prompted with context and reply. On MixSarc this kind of model peaked at 0.3905 sarcasm F1 (LLaMA 4 Maverick, reported in MixSarc).

Two ablations isolate the key ingredients.

  • Unpaired contexts. Synthetic contexts are generated without their label-flipped partner, at equal count. This tests whether the paired structure is the active ingredient.
  • No verifier. All generated contexts are kept. This tests whether the filtering is doing work.

Fairness rules. Every arm gets the same encoder, the same number of training steps, and the same 20-trial hyperparameter search. Every augmented arm gets the same synthetic-example budget. Every arm is scored at the natural sarcasm rate.

Why not just…?

Why not just reweight the loss or tune the threshold? This is the most plausible threat. On a dataset of about 3,000 tweets, a well-tuned weighted loss may capture most of the available gain. The direct comparison with the imbalance baseline settles it. If reweighting matches context-swap augmentation within seed variance on the natural-ratio test set, the augmentation story fails on that dataset and we will say so. Reweighting cannot, however, supply context. The context-dependent evaluation described below is where we expect the two to separate.

Why not just use token-level augmentation as akaBERT did? Word substitution keeps the label tied to surface cues in the reply. It cannot create an example whose label depends on what came before. Its reported jump from 77% to 93% is not on the official sarcastic-class metric, so we will re-measure it under that metric. If it still matches our method on context-free test sets, our advantage must appear on the context-dependent sets or not at all.

Why not just prompt a large language model with the context? MixSarc reports a best zero-shot sarcasm F1 of 0.3905, with very low recall from the other models. Hong et al. report that GPT-4-turbo misses innuendo and unexpected twists despite roughly 88% overall accuracy. We include the zero-shot reference anyway, run on the same contexts, because it also tells us whether the generator "understands" the sarcasm it is asked to stage.

Why not just generate whole sarcastic dialogues, replies included? This is the Hong et al. approach, and it is a real competitor. Our bet, grounded in Groshan et al. and in MixSarc's observation that translation and generation flatten code-mixed style, is that synthetic punchlines teach unnatural cues in low-resource varieties. The reply-rewriting baseline at equal budget tests exactly this. If it wins, the authenticity argument is wrong.

Proposed Experiments

The decisive controlled test uses English, where real contexts exist. We take the FigLang2020 Twitter data, on which Jaiswal reports 0.790 F1 with three-turn context and Lee et al. report 0.931. We subsample the training set to iSarcasmEval-like sizes (a few hundred sarcastic threads at a 1:3 ratio) and strip the real contexts. We then compare training on three versions: real contexts, synthetic contexts produced by our pipeline, and no contexts. The test set is real threads subsampled to a 1:4 ratio. The measure is the share of the real-context gain that synthetic contexts recover. If they recover little, the method has no basis and we stop.

The low-resource evaluation covers Arabic and Bangla-English. For Arabic iSarcasmEval and MixSarc we report the sarcastic-class F1 on the official test sets, plus AUC (area under the ROC curve), which Abercrombie and Hovy show is stable when the class ratio shifts. Because both benchmarks are context-free, we also build a small evaluation set of a few hundred real Arabic and Bangla-English comment threads, annotated by native speakers at the sarcasm rate they naturally show. A context-free result alone cannot tell us whether the model learned to use context. The thread set can.

Pair sensitivity is a direct behavioral test. We hold out a native-speaker-written set of context-swapped pairs and measure how often the detector flips its label when only the context changes. A model that reads only the reply scores at chance on this test, whatever its aggregate F1.

Finally, a filtering audit. We report human agreement with the verifier's accept and reject decisions. We also report F1 as a function of how strictly the filter is set, which shows whether verification, rather than sheer volume, drives the result.