Finding Where Sarcasm Lives in Speech: Prosodic Surprise Decoded as Spans

Spoken sarcasm usually rides on a few words. "Oh, great, another meeting" is sarcastic because of how one word is said: stretched, flattened, or pitched against what the sentence would normally call for. Current multimodal sarcasm detectors throw that structure away. They pool a whole clip into one vector and output a single yes/no label. That makes them hard to inspect, easy to fool with speaker identity and keywords, and silent on where the speech contradicts the text.

We propose to recover that location without any word-level labels. The model has two parts. The first predicts, word by word, how each word would normally be spoken given its text context; we call the gap between that prediction and the actual delivery prosodic surprise. The second is a structured decoder that groups surprising words into a few contiguous "sarcasm cue" spans. The decoder is trained only from clip-level sarcasm labels, and the clip label is read off the decoded spans. In two sentences: learn a shared space in which text predicts how speech should sound, measure per word how far the real delivery departs from that prediction, and let a sequence model with a contiguity prior turn those departures into a small number of spans that both explain and decide the label. If this works, it gives the first word-level localization of spoken sarcasm and a detector whose evidence is tied to the speaker's own delivery rather than to who the speaker is.

Core Claim

Our central claim has three parts. First, a word-level contrastive alignment between text context and prosody yields a per-word incongruity signal that existing utterance-level audio features do not provide. Contrastive alignment here means training two encoders so that matching text and speech land close together in one embedding space. Prosody means pitch, duration and loudness. Second, decoding that signal with a latent conditional random field gives localizations that are more accurate and more compact than reading attention weights off a pooled classifier. A conditional random field (CRF) is a sequence model that scores whole label sequences, so it can prefer a few contiguous spans over scattered single words. "Latent" means the word tags are never observed and are summed over during training. Third, because prosodic surprise is computed relative to the speaker's own text, the resulting detector should lose less accuracy when tested on unseen speakers.

The weakest of these claims, and the one the experiments are built to break, is the second. A pooled classifier with attention may localize just as well. If it does, we drop the CRF and report the prosodic-surprise signal on its own.

Relation to Prior Work

Three groups of prior work shape this design. The first motivates it: no existing sarcasm model localizes. Multimodal sarcasm work on video and speech pools to one label. That includes the original MUStARD study (Towards Multimodal Sarcasm Detection), which fuses BERT, ResNet and MFCC (mel-frequency cepstral coefficient, a classic hand-built audio feature) inputs in an SVM. It also includes the encoder-upgraded MUStARD++ Balanced pipeline (Sarcasm in Sight and Sound), which uses BART text, ViFi-CLIP video and a Wav2vec 2.0 model fine-tuned for speech emotion recognition, joined by gated attention. MuLOT (Multimodal Learning using Optimal Transport) replaces cross-attention with optimal-transport kernels, a parameter-light way to match features across modalities, to cope with MUStARD's fewer than 700 clips. HCT-DMG (Cross-Attention is Not Enough) gates modalities hierarchically to handle conflicting signals. All four output one pooled decision. We keep their fusion insight but replace the pooled head with spans.

The image-text debiasing work also constrains us. DMSD-CL (Debiasing Multimodal Sarcasm Detection with Contrastive Learning) and MICL (Multi-View Incongruity Learning) show that sarcasm detectors latch onto keywords. DMSD-CL reports that prior cross-attention models fall to roughly 34–37% accuracy on an out-of-distribution test set, while its own counterfactual contrastive training holds at 70.25%. We borrow their counterfactual logic: change one factor, check whether the prediction flips. We apply it to audio rather than images.

The second group supplies the incongruity signal. CLAPSpeech (CLAPSpeech) contrastively aligns text-context encodings with pitch and duration at the word and phoneme level, while filtering out speaker timbre. It does this for text-to-speech, where it raised LJSpeech MOS (mean opinion score, a listener rating of naturalness) from 3.77 to 4.11. Speech-CLAP (Speech-CLAP) aligns whole utterances with style descriptions (Pearson 0.70 on its benchmark), but only at the clip level. We reuse CLAPSpeech's word-level objective for the opposite purpose. Instead of generating expected prosody, we score how far observed prosody departs from expectation. The research file notes that fine-grained temporal alignment of "localized intonation drops signaling irony" is open, and that is the gap we target.

The third group supplies the decoder. CRFs over neural features prevent fragmented and illegal tag sequences in text tagging (Structured prediction models for RNN based sequence labeling in clinical text; DeepVar; Named Entity Recognition for Chinese biomedical patents). Those models are always trained on gold token tags. Ours never sees token tags: the CRF is a latent variable trained through the clip label. One caution from that literature carries over directly. Transition matrices can over-constrain outputs when boundaries are fuzzy, and sarcasm-cue boundaries are fuzzy. The parallel Bregman CRF (Bregman Conditional Random Fields) is not needed here, because sitcom utterances are short.

The closest sequence-learning sarcasm system, MarSan (MarSan at SemEval-2022 Task 6), runs a BiLSTM over T5 features but pools into a softmax, works only on text, and does no tagging.

Research Program

Aligner. We first train a CLAPSpeech-style aligner on read speech, following that paper's LibriSpeech setup, with words time-stamped by forced alignment. The text side encodes word $w_i$ in its sentence context as $f_i$. The prosody side encodes the pitch, duration and energy contour of the aligned audio segment as $g_i$, with speaker timbre removed as in CLAPSpeech. A symmetric contrastive loss pulls each word's text vector toward its own prosody vector and away from other words' prosody vectors in the batch.

Prosodic surprise. For each word in a target utterance, prosodic surprise is

$$s_i = 1 - \cos(f_i, g_i).$$

Here $s_i$ is high when the word was delivered unlike the way the text context predicts. Because $g_i$ is computed with timbre filtered out and $f_i$ depends only on text, $s_i$ should carry little speaker identity. We test that directly.

Emissions. The emission network gives each word scores over three tags: outside a cue, begin a cue, inside a cue. Its inputs are $s_i$, the raw prosody vector $g_i$, and contextual text features that include the preceding dialogue turns. Gating between text and prosody follows HCT-DMG's observation that unweighted fusion lets text dominate.

Decoder and clip label. A linear-chain CRF with a learned transition matrix sits on top. The matrix is initialized to favor few, contiguous spans. A clip counts as sarcastic exactly when its tag sequence contains at least one cue:

$$P(\text{sarcastic} \mid x) = 1 - \frac{\exp \mathrm{score}(O,\dots,O \mid x)}{Z(x)}.$$

Here $\mathrm{score}$ is the summed emission and transition score of a tag sequence, and $Z(x)$ is the sum over all tag sequences, computed by the forward algorithm. Training minimizes cross-entropy on this probability. Localization at test time is the Viterbi path, the single highest-scoring tag sequence. A small penalty on the expected number of cue words keeps a sarcastic clip from being explained by tagging every word.

Evaluation data. We need gold spans to evaluate localization. We will annotate the sarcastic clips of MUStARD++ Balanced. Three annotators will listen to each clip and mark the words whose delivery carries the sarcasm. We will report inter-annotator agreement and treat low agreement as a finding, given the perception gap documented by iSarcasm.

As a secondary, text-only check of the latent decoder, we will derive silver spans from iSarcasm's author-written non-sarcastic rephrasals. The silver span is the set of words changed between the sarcastic tweet and its sincere rewrite. This tests whether the latent CRF finds the inverted words when only text emissions are available.

Baselines

The naive reference points are majority-class prediction for detection, and two simple localizers. One marks the word with the largest raw pitch excursion; the other marks the most sentiment-laden word from a lexicon. These show how much of "localization" is trivial.

The strongest existing detector is the trimodal context model of Sarcasm in Sight and Sound. It combines BART text, emotion-tuned Wav2vec 2.0 audio and ViFi-CLIP video and reports a macro-F1 of 0.736 on MUStARD++ Balanced. It is the strongest competitor because it already uses a modern pretrained speech encoder and is evaluated on the same data. We will re-run it on our exact folds with the same tuning budget, and the same number of seeds as our model. We also report MuLOT's reported 76.82% accuracy on MUStARD (Multimodal Learning using Optimal Transport), and the speaker-independent text-plus-audio figure of 63.1% F-score from Towards Multimodal Sarcasm Detection, which is the reference for the speaker-held-out split. For localization there is no reported result found on Lacuna; every localization baseline is one we construct.

On detection, beating these means matching 0.736 macro-F1 within seed variance on the standard split. Detection gains are not the main claim. On the speaker-held-out split, beating them means clearly exceeding the re-run Bhosale model.

The key ablations isolate each ingredient. The first replaces the latent CRF with attention-based pooling over the same emissions. This is a multiple-instance classifier, a model that sees only a bag-level label and learns which items in the bag matter. Its attention weights, thresholded on a validation set, serve as localization. The second replaces prosodic surprise with the frame-pooled emotion-tuned Wav2vec 2.0 features. The third replaces the contrastive aligner with a regression model that predicts pitch and duration from text and uses its residual as surprise. All variants share data, encoders where applicable, and the tuning budget.

The proposal beats its ablations if the CRF yields higher word-level F1 and better span compactness than attention pooling, and if contrastive surprise beats both the Wav2vec 2.0 features and the regression residual on localization and on speaker-held-out detection.

Why not just…?

Why not just read attention weights off a pooled classifier? This is the most serious threat. An attention-pooled model trained on the same clip labels already produces a per-word importance score. Nothing in the file shows that attention fails at sarcasm localization; the argument for the CRF is only that its transition structure enforces contiguous, sparse spans, where attention spreads over many words. The first ablation settles this on gold spans. If attention matches the CRF on word F1 and span count, we report that and keep prosodic surprise as the contribution.

Why not just use the emotion-tuned Wav2vec 2.0 features that already gave a 2% gain? Those features describe how the clip sounds in absolute terms. They do not say whether a word sounds the way its text predicts, so a naturally expressive speaker looks sarcastic all the time. That explanation fits the "speaker effect" behind the drop from 71.6% to 63.1% F-score between speaker-dependent and speaker-independent setups in Towards Multimodal Sarcasm Detection. The second ablation, together with a speaker-identification probe on each feature type, tests whether surprise carries less speaker identity and generalizes better.

Why not just predict prosody from text with a regression model and take the residual? This is a genuine alternative, and contrastive training may not be needed. CLAPSpeech's MOS gain over a prediction-based baseline (3.77 to 4.11) suggests contrastive targets capture context-dependent prosody better than regression to a mean contour, which matters because sarcastic delivery is the tail of the distribution. That evidence comes from speech synthesis, not detection. The third ablation decides it.

Why not just have a large language model point at the sarcastic words in the transcript? Text-only localization will find "great" in "oh, great." It will miss cases where the words are neutral and only the delivery inverts them, and it inherits the keyword shortcuts documented by DMSD-CL. We include a text-only emission variant and break results down by whether the gold span contains a sentiment word. The proposal is only worth its cost if it wins on the neutral-word cases.

Proposed Experiments and Decisive Checks

Detection is evaluated on MUStARD++ Balanced under two protocols: the standard split, and a speaker-held-out split in which every clip from a given speaker is in test or in training, never both. Localization is evaluated on our gold spans with three measures: word-level F1, exact and overlap span F1, and a pointing-game score, meaning whether the single highest-scoring word falls inside a gold span. The iSarcasm silver-span test reports the same measures for the text-only latent CRF.

The decisive check is a prosody-flattening counterfactual, in the spirit of DMSD-CL's rewrites. We will resynthesize each test clip with a vocoder, keeping words and timing but flattening pitch and equalizing per-word duration. If the model relies on delivery, the sarcastic probability should fall, and the CRF's cue spans should disappear most on clips whose gold spans contain no sentiment word. A text-shortcut model would be unchanged.

We also run the reverse probe. A speaker-identification classifier trained on prosodic surprise should perform near chance, while one trained on the Wav2vec 2.0 features should not.

The idea fails cleanly if any of three things happen: flattening leaves predictions unchanged, attention pooling matches the CRF on localization, or regression residuals match contrastive surprise. Each outcome tells us which component to drop.