Sarcastic Tone Is Relative: Speaker- and Turn-Referenced Prosody for Spoken Sarcasm Detection

Spoken sarcasm detectors mostly listen to one utterance at a time. They take the audio of the sarcastic line, extract pitch, energy, or learned speech embeddings from it, and ask whether it "sounds sarcastic." This proposal argues that the question is badly posed. A flat, slow "oh, great" is only deadpan if this speaker is usually lively, or if the previous speaker was excited. The same acoustics from a naturally monotone speaker, or after a subdued turn, mean nothing. We propose to represent sarcastic delivery as a deviation: from the speaker's own habitual voice, and from the prosody of the turn it answers. A classifier built on these relative features should recover acoustic signal that current systems discard when they treat each utterance in isolation.

The claim is sharp enough to fail, and the literature already contains the puzzle it would explain. In telephone speech, hand-picked pitch and energy features taken from the target "yeah right" alone were unreliable, and adding them to spectral features lowered accuracy (“yeah right”: sarcasm recognition for spoken dialogue systems). If sarcastic tone lives in contrast rather than in absolute values, that negative result is what one would expect, because absolute prosody mixes the sarcasm signal with differences between speakers. The same hypothesis predicts where current multimodal systems break: on clips shorter than two seconds, which give a model no internal reference point (Sarcasm in Sight and Sound).

Core Claim

Consider a frozen speech encoder $h(\cdot)$, for example Wav2vec 2.0 fine-tuned for speech emotion recognition as in Sarcasm in Sight and Sound. Let $u_t$ be the target utterance by speaker $s$ and $u_{t-1}$ the turn it responds to. We form two contrast vectors:

$$ d_{\text{spk}} = h(u_t) - \frac{1}{|B_s|}\sum_{u \in B_s} h(u), \qquad d_{\text{turn}} = h(u_t) - h(u_{t-1}), $$

where $B_s$ is a pool of unlabeled non-target utterances by the same speaker, drawn from outside the evaluated clip. These vectors are passed to the classifier together with $h(u_t)$ and the text. The claim is that $d_{\text{spk}}$ and $d_{\text{turn}}$ carry sarcasm information that $h(u_t)$ does not. The information should be largest for short clips and in the speaker-independent setting. It should be near zero for sarcasm whose incongruity depends on facts about the world rather than on delivery.

What is new here is not the use of context. Multimodal sarcasm work already feeds context turns to the model: MUStARD concatenates context features into a support vector machine (Towards Multimodal Sarcasm Detection), and MUStARD++ fuses context through gated attention (Sarcasm in Sight and Sound). What is new is that we treat acoustic context as a reference frame for measuring deviation, not as more evidence to pool. We also treat the speaker's own voice, not only the dialogue, as part of that frame. Text work supplies both motivations. Text sarcasm is an incongruity between a reaction and the situation that prompted it (Harnessing Context Incongruity for Sarcasm Detection). Preceding turns help and following turns do not (Sarcasm Analysis Using Conversation Context). Author history helps separate intended from perceived sarcasm (Exploring Author Context for Detecting Intended vs Perceived Sarcasm). We carry the incongruity idea from text to prosody, and the author-baseline idea from writing style to voice.

Research Program

The work has three parts.

Build. We build the relative-prosody classifier. The encoder is frozen, so any gain comes from the representation and not from extra acoustic training. Speaker pools $B_s$ come from audio the benchmarks already contain. PodSarc offers hours of the same podcast hosts (Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection). MUStARD and MUStARD++ provide many other clips of the same sitcom characters. No label from a pool utterance is ever used. Pool utterances never come from the test clip's conversation, and in the speaker-independent split they never come from training clips.

Analyze. We test the mechanism rather than just the headline number. Three analyses do this: gains broken down by clip duration, gains broken down by the four MUStARD++ sarcasm types, and a replay of the "yeah right" finding. The replay compares raw and speaker-normalized versions of the same hand-crafted pitch and energy features.

Scope with external knowledge. The material shows that some sarcasm cannot be resolved from local conversation or sensory cues at all. Such cases depend on facts about the world or social norms (Transformer-based Context-aware Sarcasm Detection in Conversation Threads from Social Media; Ambiguity-aware Multi-level Incongruity Fusion Network). Text methods address this by retrieving commonsense, as in Detecting Emotional Incongruity of Sarcasm by Commonsense Reasoning and Affective Dependency Graph for Sarcasm Detection. We do not build a knowledge module. Instead, annotators label a test slice by whether a human needs outside facts to see the sarcasm. We then check the prediction that relative prosody helps on the "delivery-marked" slice and not on the "knowledge-marked" slice. If that holds, it tells future systems which signal to route where. This is the kind of division that the modular agents of Commander-GPT assume but do not measure for speech.

Baselines

Naive reference point. The first baseline is a target-only bimodal classifier that uses the same frozen encoder, the same text encoder (BART), and the same fusion head, but no context and no speaker pool. It is the isolated-utterance setup the proposal criticizes. Every other system is compared against it with identical data splits, five seeds, and the same hyperparameter search budget.

Strongest existing method on MUStARD++. This is the trimodal context-aware model of Sarcasm in Sight and Sound: BART text, Wav2vec 2.0 speech-emotion audio, and ViFi-CLIP video, fused with context through collaborative gated attention. It reports a macro-F1 of 0.736 on MUStARD++ Balanced. It is the strongest competitor because it already sees context audio. If explicit contrast is only a reparametrization of what gated attention learns, this model will match ours.

We run it two ways under matched tuning. The first run reproduces it as published. The second run appends our $d_{\text{spk}}$ and $d_{\text{turn}}$ to its audio stream. Beating it means a macro-F1 gain across seeds, with the gain concentrated in clips under two seconds.

Other published reference points. On PodSarc, the published bimodal model reports an F1 of 73.63% with human-verified labels (Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection). This is the reference for the setting where speaker pools are richest. On the original MUStARD speaker-independent split, text plus audio reached a 63.1% F-score (Towards Multimodal Sarcasm Detection). An audio-only VGGish transfer model reported a 72% F-score on MUStARD (Deep CNN-based Inductive Transfer Learning for Sarcasm Detection in Speech); the file does not say which split that number uses, so we re-run it under our splits before comparing. Zero-shot GPT-4o reports a macro-F1 of 67.47% on MUStARD++ (Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection). We re-run it with the same context turns as text for a direct comparison.

Ablations that isolate the key ingredient. - Capacity-matched concatenation. A model receives $h(u_{t-1})$ and the speaker-pool mean as extra inputs, rather than their differences, with the same parameter count. - Speaker-only and turn-only arms. These separate the two reference frames. - Shuffled-context control. $u_{t-1}$ is swapped for a turn from an unrelated conversation. If $d_{\text{turn}}$ still helps, the benefit comes from general normalization rather than from conversational contrast. - Normalized hand-crafted prosody. Pitch and energy, z-scored within speaker, fed to a small classifier. This tests whether the idea needs learned features at all.

Why not just…?

Why not just prompt a large multimodal model with the whole conversation? The evidence in the file says monolithic zero-shot models underperform here. GPT-4o reached a macro-F1 of 67.47% on MUStARD++ and needed human phonetic checking on subtle prosody (Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection). A fine-tuned 110M-parameter commander beat zero-shot GPT-4o in Commander-GPT. We still run the model with context turns, so the comparison is measured rather than assumed.

Why not just concatenate the context audio and let attention learn the contrast? This is the real threat. Gated context fusion already exists, and a flexible model can in principle compute differences on its own. There are two reasons to doubt it will do so in practice. First, the datasets are small: 690 MUStARD clips plus 164 added clips in MUStARD++. Second, gating studies in speech recognition found that naive context concatenation overfits as history grows (Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion). The capacity-matched concatenation ablation settles the question directly. If it matches the contrast model, we will report that relative prosody is learnable from raw context and that the explicit form adds nothing. The duration and speaker-normalization analyses remain informative either way.

Why not just z-normalize classic pitch and energy per speaker? If this cheap version works as well as learned contrast features, the core hypothesis is confirmed and the method becomes simpler. That would be a good outcome, not a failure. We therefore run it as a first-class arm and a direct replay of the "yeah right" negative result (“yeah right”: sarcasm recognition), rather than treating it as a straw man. The file also shows that learned self-supervised features beat classic features overall (Sarcasm in Sight and Sound; Deep CNN-based Inductive Transfer Learning), so we expect the learned contrast to win on absolute score. We would not be surprised if the relative gain were similar.

Why not just add commonsense retrieval instead? Retrieval addresses a different failure: sarcasm that clashes with world facts. EICR-style reasoning (Detecting Emotional Incongruity of Sarcasm by Commonsense Reasoning) has only been shown on text. It cannot hear a deadpan reply to an enthusiastic question. The knowledge-slice analysis tests whether the two signals cover different cases. If relative prosody also helps on the knowledge-marked slice, our story about where its signal lives is wrong.

Decisive Checks

Four results would support the proposal, and each has a clear failing counterpart:

  1. Duration. The gain over the target-only baseline should grow as clips get shorter, largest below two seconds. A flat or reversed trend would argue against the reference-frame explanation.
  2. Speaker normalization in the replay. Speaker-normalized prosody should turn the "yeah right"-style harm into a benefit on MUStARD++ and PodSarc.
  3. Real conversational contrast. The shuffled-context control should lose most of the $d_{\text{turn}}$ gain. If it does not, the turn benefit is ordinary normalization.
  4. Knowledge slice. Gains should vanish on the knowledge-marked slice.

If the capacity-matched concatenation matches our model on all four, the explicit representation is unnecessary. What would survive is the empirical finding that sarcastic tone is relative. If none of the four hold, the hypothesis is wrong, and isolated-utterance acoustics were never the bottleneck.