Forecast Before You Read: Separating Deadpan Sarcasm from Genuine Neutrality

Sentiment systems fail most often on sentences that look emotionally flat. A tweet like "Great, the merger closed" or a radio guest saying "Well, that went as planned" may be a sincere neutral report, or it may be sarcasm delivered with a straight face. The words alone cannot settle it. What settles it is expectation: given what this speaker has been saying, and what just happened, did we expect a strong opinion here? Human readers form that expectation before they parse the sentence. Current automatic judges do not. They read the target sentence first, and its literal wording then anchors everything they conclude.

We propose to make the expectation explicit. A forecaster sees only the speaker's history and the surrounding conversation, never the target utterance, and outputs a probability distribution over what stance the next utterance will take. We borrow the multi-mode setup from vehicle trajectory prediction, where a model predicts several possible maneuvers with a probability for each. Only after committing to that forecast does a large language model acting as a judge ("LLM-as-a-judge", a model prompted to score or classify another text) read the target and decide among sincere-polar, sincere-neutral, and sarcastic. The central claim is that the gap between the forecast stance and the literal stance carries most of the signal for deadpan sarcasm. We also claim that forcing the forecast to happen blind, before the target is visible, is what makes the judge use that signal, not just having the context in the prompt.

Why this gap, and what already exists

Three findings motivate the proposal. iSarcasm collected sarcasm labels from the authors themselves. It found that third-party human annotators miss 30% of intended sarcasm, falsely flag 45% of sincere tweets, and reach only 0.616 F-score. Models trained on hashtag data fall from 0.874 to 0.364 F1 on the same data. An LLM judge is also a third party, so it should inherit these blind spots. DiPlomat showed that in multi-turn dialogue the hard step is noticing that a turn is non-literal at all, with ALBERT reaching 65.1% on turn identification. It also found that RoBERTa-large did worse than RoBERTa-base, which the authors attribute to stronger literal anchoring in larger models. Work on neutrality explains why deadpan cases are hardest. IIIT-H at SemEval 2015 and Sentiment Analysis: It's Complicated! show that the neutral label lumps objective statements together with balanced or suppressed opinion. As a result, classifiers default to neutral exactly when real tension is present. Will-They-Won't-They shows the same confusion for stance: its neutral Comment class is routinely mistaken for Support.

The closest existing methods each hold one piece of the idea. Exploring Author Context for Detecting Intended vs Perceived Sarcasm conditions a classifier on an author's past posts. We reuse that idea of a per-author baseline, but we turn it into an explicit forecast that is kept separate from the target, and we evaluate it specifically on neutral-looking inputs. TRACE has a judge weigh benign against adversarial hypotheses over a multi-turn history. It targets jailbreak escalation, and its judge always sees the full conversation, target included. PASTEL shows the value of splitting work between a small model and a verifying judge: a fine-tuned 1B extractor plus a GPT-4o filter reaches 80.87 F1 on 14Res, against 52.54 for few-shot chain-of-thought GPT-4o. PASTEL works on isolated sentences and takes literal polarity at face value. We keep its division of labor, a small specialized model feeding a judge, but the small model's job becomes forecasting stance, not extracting spans.

From trajectory prediction we take two design choices and nothing more:

  • Several possible stances. Following Messaoud et al., which assigns attention heads to distinct maneuvers, the forecaster outputs several stance modes with probabilities instead of a single point estimate.
  • Predicting the change. Following motion-increment prediction, the forecaster separates the speaker's stable baseline stance from a predicted change caused by the latest event or turn.

We stop there on purpose. The LLM-in-driving papers, context-aware MTR (+0.95% mAP on WOMD) and CoT-Drive, put the language model upstream of a numeric forecaster. We reverse that order and put the forecaster upstream of the language model. Their finding that GPT-4V predicts intention less accurately than a specialized model is part of why the forecaster here is a small, trained model and not a prompted LLM.

Core Claim

Let $\hat{p}_t$ be the forecaster's distribution over stance classes (negative, neutral, positive) for speaker $u$'s utterance at turn $t$, computed from history alone. Let $q_t$ be a literal-sentiment classifier's distribution over the same classes for the utterance itself. The expectation-violation signal is

$$r_t = \mathrm{KL}(\hat{p}_t \,\|\, q_t),$$

the Kullback–Leibler divergence, which is large when the literal reading assigns low probability to stances the forecast considered likely.

We predict that on utterances that look neutral on the surface, $r_t$ separates deadpan sarcasm from sincere neutrality much better than anything computable from the utterance alone. Sincere neutrality should show low $r_t$, either because the speaker is predictably flat or because the forecast was itself uncertain. Deadpan sarcasm should show high $r_t$ together with a confident polar forecast. The judge receives $\hat{p}_t$, $q_t$, and $r_t$ along with the text. It is instructed to reconcile them under two competing hypotheses: sincere, or inverted meaning.

Research Program

Forecaster. The forecaster is a small encoder, RoBERTa-base, chosen because of the DiPlomat scale anomaly. It has two branches. The first encodes the speaker's baseline from long-range history: past posts for iSarcasm, the author's earlier tweets about the same deal for WT-WT, earlier turns by the same speaker for DiPlomat. The second encodes the local change from the immediately preceding turn or event. The combined output is a mixture over stance modes. Training labels come from a literal-sentiment tagger applied to each speaker's non-target utterances, so no sarcasm labels are needed to train it. When history is missing, as with short or new author timelines, the forecaster must output wide, uncertain distributions. We check its calibration directly with expected calibration error on held-out turns.

Judge. The judge is an instruction-tuned LLM, run at three sizes to test whether larger judges are more literal, as DiPlomat's result suggests. It uses a fixed two-hypothesis prompt and receives the forecast summary only after the forecaster has run blind.

Diagnostic subset. We build a "neutral-surface" evaluation subset by combining two filters: a literal-sentiment tagger's neutral output, and iSarcasm's author-provided literal rephrasals. This isolates the deadpan cases where the claim should hold. All headline metrics are also reported on this subset, because gains on overtly sarcastic tweets like "love getting a flat tire" are uninformative here.

Baselines

Naive reference. The do-nothing reference labels every item non-sarcastic, or uses the literal-sentiment tagger alone. On iSarcasm's 1:5 class balance this sets the floor for sarcastic-class F1 and shows how much any method gains over literal reading.

Strongest reported numbers. The strongest published figure in our file is the 0.364 F1 from iSarcasm, alongside the 0.616 third-party human F-score, which serves as a practical target. For DiPlomat turn identification the reference is ALBERT's 65.1% accuracy DiPlomat. For WT-WT it is SiamNet's 62–63% macro F1 against a 75% human bound WT-WT.

Author-context classifier. We re-run the author-context model of Oprea et al. on our splits with the same history window. No reported result on iSarcasm was found on Lacuna for this model, so it must be reproduced. It is the strongest competitor that uses history without a forecast.

Context-in-prompt judge. This is the most important baseline: the same LLM judge given the identical history and target text in one prompt, with the same token budget, the same two-hypothesis instructions, and the same number of prompt-tuning iterations on the development set, but with no separate forecast.

Ablations. Three ablations isolate the key ingredient. First, feed the forecast from a mismatched author's history; any gain that survives this is not using the speaker's history. Second, compute the forecast after the target is visible, which breaks the blindness. Third, drop the judge and threshold $r_t$ directly, which tests whether the LLM adds anything beyond the divergence itself.

What counts as winning. The proposal beats the field only if it exceeds both the author-context classifier and the context-in-prompt judge in sarcastic-class F1 on the neutral-surface subset. The margin must be larger than bootstrap confidence intervals over five seeds.

Why not just…?

Why not just put the author's history in the judge's prompt? This is the real threat. A strong LLM might form its expectation implicitly. The evidence against it is indirect. PASTEL shows that frontier models do much better as verifiers of structured intermediate outputs than as end-to-end reasoners, and DiPlomat suggests that literal anchoring grows with scale. The context-in-prompt baseline and the post-hoc-forecast ablation settle this directly. If either matches the blind forecast on the neutral-surface subset, the claim is false and we will report it that way.

Why not just fine-tune an author-context classifier? Oprea et al. already do this, but a single classifier has no mechanism to tell "this speaker is predictably flat" apart from "this speaker should be angry here and is not." We therefore expect it to repeat the default-to-neutral failure documented by Chikersal et al. on deadpan items. Comparing the two on the neutral-surface subset, with shared history, tests this. If the classifier matches us, the judge is unnecessary and the contribution shrinks to the diagnostic subset.

Why not just threshold the surprise score and skip the LLM? It may work on WT-WT, where stance is closer to a scalar quantity. It likely fails on DiPlomat, where hyperbole and idiom also produce high surprise without being sarcastic. The judge-free ablation measures exactly this, per dataset.

Decisive Checks

We run on iSarcasm, the DiPlomat identification task, and WT-WT, with Comment-versus-Support confusion as the neutrality metric on WT-WT.

The idea stands if two things hold on the neutral-surface subset. First, the forecast-then-judge system beats the context-in-prompt judge. Second, that gain shrinks when the forecast comes from a mismatched author or is computed after seeing the target.

A further prediction should hold. If literal anchoring is the mechanism, the gain from blind forecasting should be largest for the largest judge. If the gain is flat across judge sizes, or appears only on overtly sarcastic items, the expectation-violation account is wrong, even if the aggregate F1 improves.