Spoken sarcasm often lives in tone of voice: "Oh, great" said flatly after bad news means the opposite of its words. Current audio-text models handle this badly. A recent diagnostic study found that large speech-text models flag sarcasm whenever they hear high pitch and irregular pauses. Yet genuine English sarcasm in sitcom data tends to have a lower pitch register. Simply editing the pitch and pauses of sincere utterances made these models raise false alarms on up to 60% of them. We propose a different input signal. A word should count as a sarcasm cue only when its pitch, loudness, length, or following pause departs from what a listener would expect for that word in that position from that speaker. How loud or high the word is in absolute terms should not matter. This follows a long line of prominence research showing that listeners judge emphasis by contrast, not by absolute level.
Concretely, we train a text-conditioned model of expected prosody on ordinary, non-sarcastic conversational speech. We score every word of a test utterance by how surprising its observed prosody is under that model. A small latent-state conditional random field (a sequence model that scores whole chains of hidden word states jointly, rather than each word on its own) then reads these surprise scores together with the words. It outputs an utterance-level sarcasm decision and a per-word probability that the word carries the cue. The claim is testable in two ways. First, detectors that use prosodic surprise should be much less fooled by pitch edits that leave the text-prosody relation intact. Second, they should match or beat absolute-prosody detectors on held-out speakers.
Core Claim and What Is New
The core mechanism comes from relative acoustic salience. Calhoun frames prominence as unexpected prominence relative to the baseline that syntax and part of speech set up. Brenier et al. show that normalizing within intonational phrases isolates contrastive emphasis better than global speaker normalization. Neither applies this to pragmatic meaning. Sarcasm is the application where the idea matters most, because the cue is exactly a mismatch between lexical content and delivery.
Prior sarcasm work falls into three camps.
- Utterance-level pooling. Castro et al. built MUStARD and classified pooled MFCC and spectral averages with an SVM. MuLOT aligns MFCC, video, and text streams with optimal transport. HCT-DMG gates modalities per utterance to handle cross-modal incongruity. All three treat audio as an absolute signal and produce one label per clip. We do neither.
- Diagnosis without a fix. Chen et al. motivate this proposal: they diagnose the expressive-prosody heuristic but propose no remedy. Their pitch-editing probe (PSOLA, a standard method that changes pitch and timing while keeping the words intact) becomes our main robustness test.
- CRFs for sarcasm, without audio. MSTI uses a BiLSTM-CRF to tag the target of mockery in tweets. It has no audio, and it locates the target rather than the cue.
The closest structural precedent is the syllabic prominence CRF/LDCRF study. It found that latent-dynamic CRFs beat plain CRFs when prominent events are sparse among long runs of non-prominent units. That is the same sparsity we expect for sarcasm cue words. However, it used read speech and acoustic features only. We reuse its latent-state design on spontaneous dialogue, with text-conditioned features.
Research Program
Expected-prosody model. We force-align conversational podcast speech of the kind SPECTRA pre-trained on (Yu et al.). For each word we extract log mean F0, F0 slope, energy, nucleus duration, and following pause. The model is a text encoder that sees the word, its left and right lexical context, and part of speech. It also receives a running speaker baseline: the speaker's statistics within the current intonational phrase and the preceding turn. It outputs a Gaussian mean and variance for each prosodic dimension.
This model is the constructive counterpart of Quantifying the redundancy between prosody and text. That work measures how much prosody text predicts. We use the part text does not predict. For word $t$ and prosodic dimension $d$, the surprise feature is
$$ s_{t,d} = \frac{p_{t,d} - \mu_d(w, c_t)}{\sigma_d(w, c_t)}, $$
where the terms are:
- $p_{t,d}$ is the observed value.
- $w$ is the word sequence.
- $c_t$ is the speaker-local acoustic context.
- $\mu_d$ and $\sigma_d$ are the predicted mean and standard deviation.
Two properties follow from this definition. A global pitch shift largely cancels through $c_t$. Lexical tone in Mandarin is absorbed by $\mu_d$, so the feature is not confused by tone the way absolute F0 is.
Latent-state CRF. Each word gets an emission vector built from its contextual text embedding, its surprise vector $s_t$, and the left/right surprise deltas of Bernhard et al.. A hidden-state CRF assigns each word one of $H$ latent states, split into states reserved for sarcastic utterances and states reserved for sincere ones. The utterance probability is
$$ P(y \mid X) = \frac{\sum_{\mathbf{h} \in \mathcal{H}_y} \exp\big(\sum_t E(h_t, \mathbf{x}_t) + \sum_t A_{h_t,h_{t+1}}\big)}{Z(X)}, $$
where the terms are:
- $\mathcal{H}_y$ is the set of latent paths permitted for label $y$.
- $E$ is the neural emission score.
- $A$ is the learned transition matrix.
- $Z(X)$ sums over all paths.
Training uses only utterance labels. Forward-backward inference gives each word's posterior of being in a "cue" state, which serves as the localization output. With $H$ around 6, the CRF adds only tens of parameters. This matters on a 690-clip corpus.
Data handling. Sitcom laugh tracks and crosstalk break forced alignment, which is the alignment penalty noted in the prosodic-integration literature. We therefore mark words with low alignment confidence and mask their surprise features rather than drop the utterance.
Baselines
The naive reference is a majority-class predictor plus a fine-tuned text-only RoBERTa on the target utterance with its dialogue context. Any acoustic method must beat text-only to justify itself.
The published reference for audio-text is Castro et al.'s speaker-independent Text+Audio SVM. It reports an F-score of 63.1% on MUStARD (Castro et al.). We will rerun it on our splits. The highest MUStARD number in the file is MuLOT at 76.82% accuracy (MuLOT), but it uses video. We will report it as published and also retrain an audio-text-only MuLOT for a fair comparison. No reported result was found for HCT-DMG on MUStARD or for any classifier F1 on MUStARD++ or MSCD. On those corpora, our baselines are our own reruns.
The strongest learned competitor is an aligned WavLM+RoBERTa model in the SPECTRA style (Yu et al.), fine-tuned end-to-end with attention pooling. It sees the same word-aligned audio as our model and can in principle learn relative cues itself. We also run Qwen-Omni zero-shot under Chen et al.'s protocol, which reproduces their false-positive figures (up to 60% on MUStARD++, up to 40% on MSCD) as the anchor for the robustness test.
Fairness. All trained systems use the same speaker-independent folds, the same context turns, 30 hyperparameter trials each, matched GPU hours, and five seeds.
Ablations isolate the key ingredient by swapping only the prosodic input to the same CRF:
- raw absolute features;
- speaker z-scored features;
- phrase-normalized features;
- local deltas;
- text-conditioned surprise.
Separately, we swap the decoder: mean pooling, attention-based multiple-instance pooling, plain CRF, and latent CRF.
What counts as beating the baselines. The proposal succeeds if surprise features plus the latent CRF do three things:
- match or exceed the best baseline's speaker-independent macro-F1 within seed variance;
- cut the PSOLA-induced false-positive rate by at least half relative to every absolute-feature system;
- localize annotated cue words better than attention weights.
Why not just…?
Why not just z-normalize pitch per speaker? This is the cheapest relative feature, and it may already remove most of the global-shift false positives. The ablation ladder tests it directly. Our prediction is that z-scoring fixes global pitch-raise edits but not local pause or pitch edits on function words. Only text conditioning knows that a long pause before "great" means something different from one before "the". If z-scoring matches surprise on both robustness and F1, the text-conditioning claim is wrong and we will report that.
Why not just fine-tune a pretrained speech-text model, or augment with PSOLA-edited negatives? This is the real threat. Counterfactual augmentation targets Chen et al.'s failure mode head on, and a fine-tuned aligned model may learn relative cues implicitly. We run augmentation both on the strongest baseline and on our model. We then test on a held-out family of manipulations not used in augmentation, such as duration lengthening and intensity edits. If augmentation generalizes as well as surprise features, the honest conclusion is that the explicit expectation model is unnecessary. Our bet rests on Chen et al.'s finding that heuristics persist even in very large models, and on the small size of MUStARD (690 clips).
Why not just attention pooling instead of a CRF? On utterance F1 alone, attention pooling may tie. We therefore claim the CRF's advantage only where structure should help: more contiguous and better-calibrated cue localization. The LDCRF result on sparse prominence (Interspeech 2012) is the evidence for this expectation. If attention matches on localization too, we drop the CRF and keep the surprise features.
Proposed Experiments and Decisive Checks
We evaluate on MUStARD (speaker-independent folds), MUStARD++ (1,202 clips), and MSCD (2,705 Mandarin clips). We report macro-F1 and false-positive rate. For localization, two annotators mark cue words on roughly 300 sarcastic utterances per language, and we measure token-level F1 and area under the ROC curve (AUROC) for the cue posteriors.
There are three decisive checks:
- Pitch-editing replication. We reproduce Chen et al.'s PSOLA probe on sincere utterances and add the held-out manipulation families. The prediction is that false-positive rates stay near the clean rate for surprise-based models and rise sharply for absolute-feature models.
- Direction of the learned cue. In English, cue states should load on negative pitch surprise over positive-valence words, consistent with Chen et al.'s observation of a lower register. A model that learned the opposite would be repeating the heuristic.
- Prosody-only condition. We repeat Chen et al.'s 300 Hz low-pass setting, where their models collapsed. Surprise features can be computed from the transcript plus the low-passed contour, so the model should retain above-chance discrimination.
References
- Predicting focus through prominence structure
- The detection of emphatic words using acoustic and lexical features
- Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
- Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection
- Cross-Attention is Not Enough: Incongruity-Aware Multimodal Sentiment Analysis and Emotion Recognition
- When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection
- Multimodal Sarcasm Target Identification in Tweets
- Investigating syllabic prominence with conditional random fields and latent-dynamic conditional random fields
- Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment
- Quantifying the redundancy between prosody and text
- Acoustic Stress Detection in Isolated English Words for Computer-Assisted Pronunciation Training