Spoken sarcasm is often signaled by how a sentence is said: a flattened pitch, a stretched word, an over-bright "great." Whether a given pitch level or speaking rate counts as marked, though, depends on the speaker. A naturally monotone speaker is not being sarcastic every time they talk flatly. Current spoken-sarcasm detectors ignore this. They compress each utterance's audio into one fixed vector and compare it against a population average. When a model is told who is speaking, it tends to memorize which characters are sarcastic by default instead of learning what sarcastic delivery sounds like. We propose to describe each word's pitch, loudness, duration and following pause as a deviation from a baseline estimated from that same speaker's earlier turns in the conversation, measured against the conversational partner's register as well. The model is told which turns belong to the target speaker. It is never told who that speaker is.
The bet is that this speaker-relative signal transfers to three settings that defeat today's models: speakers never seen in training, a different language, and the realistic regime where sarcasm is rare. Because the signal is computed word by word, it also says which words carry the sarcastic delivery. No current spoken-sarcasm system attempts this.
Core Claim
Replacing absolute, utterance-pooled acoustic features with word-level prosodic deviations from a context-estimated speaker baseline should close a large part of the gap between speaker-dependent and speaker-independent sarcasm detection. It should also transfer better from English to Spanish and degrade less when sarcastic turns are rare. Speaker identity embeddings help on familiar speakers and hurt on unseen ones; speaker-relative features should help on both.
The evidence for this gap is already on record. On MUStARD, the sitcom sarcasm benchmark, Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper) reports 71.6% F1 for speaker-dependent splits and 63.1% F1 for speaker-independent splits (Text+Audio). The paper attributes part of the speaker-dependent advantage to habitually sarcastic characters.
The motivation for normalizing relative to a speaker, rather than in absolute terms, comes from phonetics. Absolute and relative entrainment in Mandarin conversations shows that Mandarin speakers align pitch with partners only in relative register (r = 0.501), because absolute pitch carries lexical tone. Cross-linguistic boundary work shows the same durational boundary cues in English and Mandarin: Duration and pauses as boundary-markers in speech: a cross-linguistic study finds that final lengthening and pauses mark boundaries in both languages. Who Do You Think Will Speak Next? shows that listeners read de-lexicalized prosodic trajectories in both Slovak and Spanish, with asymmetries across the two languages. These studies suggest that relative, temporal cues are the part of prosody most likely to carry across languages.
What Prior Work Does and Does Not Do
Three lines of spoken and multimodal sarcasm work all classify whole utterances with pooled acoustic vectors. MUStARD concatenates Librosa descriptors with BERT and video features in an SVM. Sarcasm in Sight and Sound upgrades the audio to a wav2vec 2.0 encoder fine-tuned for emotion recognition and adds preceding turns with gated attention. Leveraging Large Language Models for Sarcastic Speech Annotation builds the PodSarc podcast corpus and an audio-text classifier. In each case, context enters as concatenated earlier turns or not at all. None of them uses earlier turns to calibrate the target speaker's voice. We keep their encoders and change only how the target utterance's acoustics are represented relative to context.
The speaker-aware dialogue models supply method ideas and one caution. Speaker-Aware BERT adds a speaker embedding to every token. TurnGPT does the same for turn-taking but uses only transcripts. We borrow their additive per-token injection but use role embeddings (target speaker versus other) rather than identity embeddings, so that nothing in the model can memorize "this character is sarcastic."
The Spanish dataset from ¡Qué maravilla! and the English-Chinese SarcNet (SarcNet) document cross-lingual gaps. However, the Spanish work trains per language, and SarcNet has no audio.
For rare-event evaluation, Putting Sarcasm Detection into Context shows that F1 collapses at a natural 1:4 sarcastic-to-literal ratio while AUC stays stable. This constrains our design: we must report both metrics and must test at realistic prevalence. Word-level localization of sarcastic cues has no spoken precedent. The closest work, Multimodal Sarcasm Target Identification in Tweets, labels the target of mockery in text with a conditional random field (CRF) and plain loss, reaching 47.9% F1. It does not label the words that carry the ironic delivery.
Research Program
For every word we compute a small prosodic vector from audio aligned to the transcript. It contains pitch in semitones, intensity, normalized duration, the pause after the word, and the word's pooled wav2vec 2.0 frames. The speaker's baseline comes from that speaker's preceding turns in the dialogue window, shrunk toward a population prior when there are few turns. The deviation for word $w$ spoken by speaker $s$ at turn $t$ is
$$d_w = \frac{x_w - \hat\mu_{s,t}}{\hat\sigma_{s,t}}, \qquad \hat\mu_{s,t} = \frac{n_{s,t}\,\bar{x}_{s,t} + k\,\mu_{\text{pop}}}{n_{s,t} + k}.$$
The symbols are defined as follows. $x_w$ is the word's prosodic vector. $\bar{x}_{s,t}$ is the mean over the $n_{s,t}$ words the speaker produced earlier in the window. $\mu_{\text{pop}}$ is the training-set mean. $k$ is a shrinkage strength tuned on validation data. $\hat\sigma_{s,t}$ is shrunk the same way. A second vector measures the target speaker's distance from the partner's recent register, which captures departures from entrainment.
Both vectors are projected and added to the text encoder's token embeddings alongside a role embedding. An utterance classifier is trained on top. A word-level head, trained by multiple-instance learning from utterance labels, produces cue scores. Multiple-instance learning here means the model is told only that a sarcastic utterance contains at least one cue word. Because cue words are a small minority of tokens, this head is where class imbalance bites. We will compare plain cross-entropy with focal loss, following Focal Loss for Punctuation Prediction with γ near 2.0, which that paper found optimal. We will also compare the self-adjusting Dice loss of IITK@Detox.
To evaluate localization, we will have three annotators mark cue words on all sarcastic MUStARD clips and on a Spanish subset. Their agreement sets the ceiling, which matters because subjective span annotation is known to be noisy.
Baselines
The naive reference point is a text-only encoder over the target utterance and its context. It tells us how much audio of any kind contributes. Alongside it we report GPT-4o's zero-shot macro-F1 of 67.47% on MUStARD++, as given in Leveraging Large Language Models for Sarcastic Speech Annotation.
The strongest existing method is the context-aware system of Sarcasm in Sight and Sound. It uses BART text, emotion-tuned wav2vec 2.0 audio and gated attention over earlier turns, and reports 0.736 macro-F1 with three modalities on MUStARD++ Balanced. Because our method concerns audio and text, we rerun its text and audio branches with identical encoders, the same context window, the same data, the same speaker-independent folds, and the same hyperparameter search budget. The file does not say whether that paper's 0.736 uses speaker-independent folds, so we compare only against our rerun, not against the published number.
On the original MUStARD speaker-independent split, the published reference is 63.1% F1 (Text+Audio, Castro et al.). MuLOT (Multimodal Learning using Optimal Transport) reports 76.82% accuracy, but its split protocol is not given in the file. On PodSarc the reference is 73.63% F1, and on the Spanish data it is 91.9% accuracy for Text+Audio.
The identity-embedding competitor is the same model with Speaker-Aware-BERT-style speaker identity vectors in place of role embeddings.
The key ablations isolate the ingredient. The first uses the same word-level features without any normalization, which tests whether word-level resolution alone explains any gain. The second normalizes with a per-speaker z-score computed over all of that speaker's audio, which tests whether in-context estimation matters. The third drops the partner-relative vector.
Beating the strongest baseline means a speaker-independent macro-F1 gain whose 95% bootstrap interval excludes zero across five seeds. It also means a smaller drop than the baseline shows when moving from speaker-dependent to speaker-independent folds, and from English training to Spanish testing.
Why not just…?
Why not just z-score each speaker's features over all their audio? This is the most serious threat, and it is the second ablation above. Global per-speaker normalization needs a quantity of each test speaker's audio that a deployed system will not have. In speaker-independent folds it becomes transductive, meaning it uses unlabeled test-set audio. The context-window estimate uses only what precedes the utterance. If the global version matches ours even with this information advantage, we will report that the gain comes from normalization itself, and the contribution shrinks to a cheaper recipe. If the context version wins, a likely reason is that speakers drift within a conversation, as the context-dependent prosodic shifts in Does Context Matter? suggest.
Why not just feed the context audio to a stronger self-supervised encoder and let it learn relativity? Bhosale et al. already pass context turns to the model, and their audio encoder is far richer than our handcrafted features. A large attention model could in principle learn to compare a speaker against themselves. We test this directly: the strongest baseline receives the full context audio, and a variant subtracts the speaker's context-mean wav2vec embedding instead of using handcrafted features. If the unnormalized encoder already closes the speaker-independent gap, the explicit normalization is unnecessary. MUStARD has only 690 clips, however, so we expect the explicit form to win in this low-data setting.
Why not just add speaker identity embeddings, which helped dialogue models? Speaker-Aware BERT's ablations show large gains for identity vectors. On sarcasm, though, identity overlaps with the label itself, which is the habitually-sarcastic-character effect Castro et al. describe. We predict that identity embeddings will raise speaker-dependent F1 and lower speaker-independent F1. The identity-embedding baseline reports both numbers, so this prediction can fail visibly.
Why not just use a text LLM with the conversation context? GPT-4o's 67.47% macro-F1 on MUStARD++ is below the audio-aware systems. The deeper limit is that text-only models cannot see prosodic markers at all, which makes them a floor rather than a ceiling for this question.
Proposed Experiments
The first decisive check is the speaker-independent gap on MUStARD and MUStARD++ Balanced. We report macro-F1 for every system on speaker-dependent and speaker-independent folds. The proposal succeeds only if our model narrows the difference while the identity-embedding model widens it.
The second check is natural prevalence. Following Abercrombie and Hovy, we subsample sarcastic test items to 1:4 and 1:9 ratios without retraining, and report F1, AUC, and the precision-recall area. The prediction is that speaker-relative features lose less F1, because a speaker's habitually flat voice no longer triggers false alarms.
The third check is cross-lingual transfer. We train on MUStARD plus PodSarc and test zero-shot and with few labeled examples on the Spanish dataset, compared against the 91.9% Text+Audio SVM trained within Spanish. That corpus consists of dubbed animation, and the file does not say whether it includes preceding turns. If it lacks them, we re-extract preceding turns from the source episodes or fall back to the population prior. We will report which case applies.
The fourth check is a tonal-language false-alarm test. There is no Mandarin spoken-sarcasm corpus in our materials. We therefore run all English-trained detectors on the cooperative Tongji Games Corpus, manually audit a sample of positive predictions, and measure how often lexical-tone pitch movement triggers a "sarcastic" prediction under absolute versus relative features. This does not show detection ability in Mandarin. It only tests whether relative features avoid the failure that the tone-constraint findings predict.
Finally, on the annotated cue-word sets, we report token F1 and exact span match against the inter-annotator ceiling for cross-entropy, focal, and Dice losses. We also report a probe of whether high-deviation words coincide with annotated cues more often than lexical-sentiment words do. If deviation-based localization does no better than sentiment-word heuristics, the claim that the model "hears" sarcasm on specific words is false, and we will say so.
References
- Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
- Absolute and relative entrainment in Mandarin conversations
- Duration and pauses as boundary-markers in speech: a cross-linguistic study
- Who Do You Think Will Speak Next? Perception of Turn-Taking Cues in Slovak and Argentine Spanish
- Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection
- Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection
- Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots
- TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog
- ¡Qué maravilla! Multimodal Sarcasm Detection in Spanish: a Dataset and a Baseline
- SarcNet: A Multilingual Multimodal Sarcasm Detection Dataset
- Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations
- Multimodal Sarcasm Target Identification in Tweets
- Focal Loss for Punctuation Prediction
- IITK@Detox at SemEval-2021 Task 5: Semi-Supervised Learning and Dice Loss for Toxic Spans Detection
- Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection
- Does Context Matter? A Prosodic Comparison of English and Spanish in Monolingual and Multilingual Discourse Settings