Hearing Sarcasm as a Departure from Syntactically Expected Prosody

Speech models that try to detect sarcasm from audio tend to treat any "expressive" delivery, such as a high pitch, a long pause or a stretched word, as evidence of sarcasm. Much of that expressiveness is not pragmatic at all. Speakers pause and reset their pitch where the grammar of the sentence tells them to. A long question with two clauses will contain pauses and pitch movement whether or not anyone is being ironic.

We propose to first predict the prosody a sentence would have if spoken sincerely, given its words and its grammatical structure. Prosody here means the melody and timing of speech: pitch, loudness, word duration and pauses. A sarcasm detector would then see only the leftover difference between what the speaker actually did and what the grammar predicts. The two-sentence version is this: a sarcasm classifier should be surprised by prosody only when the syntax does not explain it. We therefore subtract a syntax-conditioned prosody expectation from the observed audio and classify on the residual.

The opportunity exists because two literatures have each solved half of the problem. Text-to-speech research shows that pause placement and pitch resets follow from parse-tree structure. Multimodal sarcasm research shows that current systems confuse this ordinary structure with irony. The clearest case is Chen et al.. By editing only pitch and pauses on sincere utterances, they caused multimodal large language models to label up to 60% of English and 40% of Mandarin sincere utterances as sarcastic. They also found that genuine sarcastic delivery in English sitcoms has a lower mean pitch, which is the opposite of the models' internal stereotype. Nobody has asked whether those triggering pauses and pitch movements were ones the syntax already licensed.

Core Claim

A syntactic prosody model trained only on sincere speech defines, for every word, an expected pitch level, pitch range, duration, energy and following pause. Sarcastic delivery should show up as a localized, standardized deviation from that expectation. The deviation matters mostly where the text itself carries no incongruity. Measured this way, detectors become robust to grammatically licensed expressiveness without losing sensitivity to pragmatic marking.

Concretely, let $x$ be the words of an utterance, $T$ its dependency parse, and $a_w$ the observed prosodic feature vector of word $w$ after forced alignment. An expectation model with parameters $\theta$ outputs a mean $\mu_\theta(x,T)_w$ and a spread $\sigma_\theta(x,T)_w$. The residual is

$$r_w = \frac{a_w - \mu_\theta(x,T)_w}{\sigma_\theta(x,T)_w},$$

and the sarcasm detector consumes the text together with the sequence $r_1,\dots,r_n$ instead of raw acoustic features. The division by $\sigma_\theta$ matters. Positions where speakers vary a lot, such as clause boundaries, should yield small residuals even for large raw deviations.

This differs from every connecting work in the file:

  • Chen et al. diagnose the stereotype but treat prosody as a signal cut off from grammar, and build no corrective model. This proposal builds that model.
  • Li et al. find that sarcasm's prosodic signature is confined to hand-picked key phrases and differs by sarcasm type. Their study is observational and their phrases were chosen manually. We test whether residuals locate those phrases automatically.
  • The sarcastic synthesis work of Integrating Feedback Loss from Bi-modal Sarcasm Detector for Sarcastic Speech Synthesis runs in the reverse direction: a frozen detector serves as a critic for a speech synthesizer that ignores syntax. It does not improve detection.
  • Wang et al. (IAAD) use dependency structure to filter false textual incongruities in text-only sarcasm detection. We borrow that intuition, "an apparent clash only counts if structure supports it," and move it from word sentiment to acoustics.

Research Program

The expectation model. This model reuses the syntax-aware machinery of text-to-speech systems, but predicts word-level prosodic targets instead of spectrograms, which keeps it small and interpretable. The encoder follows GraphSpeech, which builds a fully connected graph over tokens from a dependency parse and encodes the path between every token pair. That design reduced spectral distortion from 8.021 to 6.821 and prosody error from 1.782 to 1.625 against a plain Transformer on LJSpeech. We add the constituency syntactic distance features of Tyagi et al., who showed that such distances help most on structurally complex questions: 8% better prosody on "wh" questions and 21% better on "or" questions. For Mandarin we use the same design with a Mandarin dependency parser. The Japanese work Applying Syntax–Prosody Mapping Hypothesis and Prosodic Well-Formedness Constraints to Neural Sequence-to-Sequence Speech Synthesis constrains our design: plain dependency distances missed some pitch phenomena that explicit phonological phrasing captured. A phrase-level variant is therefore a planned extension rather than an assumption.

Training data and leakage. The expectation model must learn what sincere conversational prosody sounds like, not read-speech prosody. Training starts on read speech such as LJSpeech and then adapts on conversational speech. This is analogous to the sitcom fine-tuning stage in the sarcastic synthesis paper, with one difference: here only utterances labeled non-sarcastic in the training folds of MUStARD++ and MSCD are used. Held-out speakers and all sarcastic utterances never touch $\theta$.

Parser robustness. Parser errors on conversational transcripts are a known weakness of syntax-to-prosody methods. We measure their effect by swapping parsers and by perturbing parses, and we report how detection degrades as parse quality falls.

The detector. The detector is deliberately simple: a pretrained text encoder whose token states are concatenated with the residual vectors, followed by a light attention layer and a classifier. Any gain should then be attributable to the residual representation, not to detector capacity. A second analysis asks whether residual magnitude is concentrated on the phrases Li et al. hand-annotated. It also asks whether residuals separate their three sarcasm types: lengthening, lower pitch and louder delivery in illocutionary sarcasm, where the words alone sound sincere; little deviation in embedded sarcasm, where the text itself contains the incongruity; and a narrowed pitch range in propositional sarcasm, which depends on surrounding context.

Baselines

Naive reference. The do-nothing points are the majority class (chance on these balanced corpora) and a text-only classifier using the same text encoder as our detector. Castro et al. report a 63.1% F-score for text plus audio in the speaker-independent MUStARD setting. That figure anchors what classical acoustic fusion adds over text alone.

Strongest published detector. This is the trimodal model of Bhosale et al.: BART for text, wav2vec 2.0 fine-tuned for speech emotion recognition for audio, and ViFi-CLIP for video, combined with gated attention. It reports a macro-F1 of 0.736 on MUStARD++ Balanced with conversational context. Because our proposal concerns audio, we retrain its text-plus-audio configuration on our exact speaker-independent folds, with the same hyperparameter search budget as our detector. We also report the trimodal number. Beating it means matching or exceeding its macro-F1 on clean data while showing a substantially lower false-positive rate under the perturbation probes described below. A clean-accuracy gain alone is not the claim, given how small the corpus is.

Large-model reference. Qwen-Omni prompted zero-shot serves as the large-model reference. We run it on the same clips and the same perturbations as Chen et al., who report up to 60% induced false positives in English and 40% in Mandarin. The file gives no numeric clean macro-F1 for these models, or for their prosody-only condition, so we measure those ourselves.

Ablations. These isolate the key ingredient, the syntax-conditioned expectation. All variants share the same detector and the same five word-level features. They differ only in how the features are prepared:

  • raw word-level features without any expectation;
  • features z-scored per speaker;
  • residuals from an expectation model that sees only word position and punctuation;
  • residuals from an expectation model that sees contextual embeddings but no parse;
  • residuals from the full syntax-aware model.

Why not just…?

Why not just normalize prosody per speaker? This is the most serious competitor. Li et al. and Chen et al. both locate the authentic signal partly in overall register (lower mean pitch, higher amplitude), and speaker z-scoring captures register cheaply. Speaker normalization cannot tell a pause at a clause boundary from the same pause inside a noun phrase, however. The licensed-versus-unlicensed probe below tests exactly this. If speaker normalization matches the full model on both clean F1 and the probe, we will report that syntax is unnecessary for this task.

Why not just let a contextual text encoder predict the expectation, skipping the parser? Pretrained encoders plausibly encode much of the relevant structure. Tyagi et al. found contextual embeddings alone competitive on compound nouns, with syntactic distance mattering on complex questions. This is a real threat to the "syntactic" half of the claim, but not to the residual idea. The parse-free expectation ablation settles it, and we will report residual accuracy broken down by syntactic complexity.

Why not just fine-tune the detector on pitch- and pause-perturbed sincere audio? Augmentation with PSOLA, the pitch- and timing-editing method Chen et al. used to fool the models, will reduce induced false positives. It does so by teaching the model to ignore pauses and pitch wholesale, which should cost recall on illocutionary sarcasm, where those cues carry the meaning. We train this augmented variant with the same budget. We then compare it with our model on two things: false positives for unlicensed (misplaced) perturbations that it never saw during training, and recall on the illocutionary subset.

Why not just use a stronger emotion-tuned speech encoder? Bhosale et al.'s plateau at 0.736 macro-F1, with errors concentrated on clips shorter than two seconds, suggests that global emotional tone is not the bottleneck. Li et al. show the sarcastic signal is local and type-dependent. The decisive evidence is the probe below: an emotion encoder has no notion of where a pause should fall.

Proposed Experiments

Corpora and metrics. We use MUStARD++ and MUStARD++ Balanced (English sitcoms) and MSCD (Mandarin stand-up), all with speaker-independent cross-validation. Across folds we report macro-F1 with confidence intervals, sincere-utterance false-positive rate, and per-type F1 on the MUStARD++ sarcasm-type annotations.

The decisive check: licensed versus unlicensed perturbation. This is a two-way perturbation probe built on Chen et al.'s PSOLA procedure:

  • Syntactically licensed edits insert pauses and pitch resets at high syntactic-distance boundaries, such as between subject and verb phrase or between clauses.
  • Unlicensed edits insert the same-sized pause or pitch jump inside a tight constituent, such as between determiner and noun.

Our predictions are:

  • the residual detector shows near-baseline false positives for licensed edits and elevated scores for unlicensed ones;
  • the multimodal large language models and the emotion-encoder baseline respond to both alike;
  • the augmentation baseline suppresses both.

If the residual model does not show this asymmetry, the core claim is wrong.

Secondary checks. The first concerns the corpus's own sarcasm types. Gains over text-only should concentrate on illocutionary sarcasm and vanish on embedded sarcasm, mirroring Li et al.'s functional trade-off. For this check we use per-type labels from the MUStARD++ annotations. The second concerns localization: we compare our residuals against Li et al.'s 114 hand-annotated key phrases. Residual magnitude should rank key-phrase words above other words in sarcastic utterances, which we measure as retrieval precision of the annotated phrases. A last test replicates the probe on MSCD. If the licensed/unlicensed asymmetry holds in Mandarin, the mechanism is structural rather than an artifact of English sitcom delivery.