Speech models that detect sarcasm tend to flag any lively-sounding voice as sarcastic. When researchers raised the pitch and inserted pauses in sincere English and Mandarin sentences, frontier speech-and-text models called up to 60% of them sarcastic. Part of the cause is that these models read pitch as raw numbers. A large share of pitch movement is fixed by the words themselves. In Mandarin, the tone of each syllable is part of its identity. In English, stressed syllables are predictably higher and longer. The question that matters for sarcasm is different: given these particular words in this language, is the voice doing something it would not normally do? We propose to answer that question explicitly. A forced aligner (a tool that, given audio and its transcript, finds the start and end time of every word, syllable and phoneme) locates each syllable. A small language-specific model, trained only on neutral speech in that language, predicts the pitch, duration and loudness the syllable would ordinarily carry. The sarcasm detector then sees only the deviation from that expectation, word by word, next to what the words mean.
In two sentences: sarcasm lives in the gap between how words are said and how those words are normally said in that language. We measure that gap per syllable with forced alignment and a monolingual "ordinary prosody" model, so the detector is not fooled by lexical tone or by generic expressiveness. The work joins four strands. Forced alignment is the core mechanism. Sarcasm detection is the application. Cross-lingual emotion recognition supplies the confound, namely that pitch means different things in different languages. Language-specific pre-training is the hypothesis about where the expectation model should come from.
Core Claim
For each aligned syllable $k$ of an utterance in language $L$, let $p_k$ be its observed prosody vector: log-$F_0$ mean and slope, duration, and RMS energy, all speaker-normalized. The detector receives the residual
$$r_k = p_k - f_L(\phi_k, \tau_k, c_k),$$
where $f_L$ is a predictor trained on neutral speech of language $L$. Its inputs are the syllable's aligned phonemes $\phi_k$, its lexical tone or stress class $\tau_k$, and its position and neighbouring syllables $c_k$. We make three claims, each testable.
First, a detector that reads residuals $r_k$ aligned to word tokens will produce far fewer false positives under pitch and pause manipulation than detectors that read raw prosody. It will do so without losing accuracy on natural sarcasm.
Second, the gain over raw prosody will be larger for Mandarin than for English, because tone accounts for more of Mandarin pitch variation.
Third, a monolingual $f_L$ will yield more useful residuals than one multilingual predictor trained on pooled languages. This extends the monolingual advantage from text models to the acoustic expectation itself.
What Exists and What Is Different
Three findings motivate the proposal. When Models Hear What They Expect probed Qwen-Omni and Gemini on MUStARD++ and the Mandarin MSCD corpus. It found that their sarcasm judgments rest on a shallow "expressive prosody means sarcasm" heuristic: PSOLA pitch and pause edits induced false positive rates up to 60% in English and 40% in Chinese, and the models had near-zero ability in prosody-only conditions. That study is diagnostic only and proposes no remedy. We use its manipulation protocol as our stress test. A Functional Trade-off between Prosodic and Semantic Cues in Conveying Sarcasm showed on hand-segmented MUStARD++ key phrases that sarcasm whose words look sincere is carried locally, through longer duration, lower mean $F_0$ and higher amplitude on key phrases, and that these effects are invisible at the utterance level. This tells us the signal is local, and its manual segments give us a check on alignment quality. Its finding that English sarcasm lowers pitch also explains why the "high pitch means sarcasm" heuristic produces false alarms. Multimodal Chinese Sarcasm Detection Integrating Audio Attributes and Textual Features found $F_0$ to be the single most important acoustic cue for Chinese sarcasm, yet it pools OpenSMILE features over the whole utterance. In Mandarin, such pooling mixes tonal pitch with attitudinal pitch.
Our method borrows from Multimodal Speech Emotion Recognition Using Cross Attention with Aligned Audio and Text. That work cut IEMOCAP audio at gold word boundaries so that the $i$-th audio chunk faces the $i$-th word in cross-attention, and this beat equal-length chunks. It used gold timestamps, English only, and basic emotions. We use automatic alignment from WhisperX, which adds language-specific wav2vec 2.0 phoneme alignment to Whisper transcripts, and from the multilingual non-autoregressive aligner in the Qwen3-ASR Technical Report. Most importantly, we do not feed aligned raw audio. We feed audio relative to a lexical expectation.
Prior work also constrains the design. Cross-lingual emotion studies show that $F_0$ conventions differ by culture: Braun et al. on dubbed speech, and Chiou and Chan on how tonal languages confuse Western-trained models. Analysis of Phonetic Level Similarities Across Languages in Emotional Speech found that emotion reshapes even identical vowels in language-specific ways, and that neutral speech aligns more closely across languages than anger or sadness. Together these suggest that a single global normalization, such as the histogram equalization of Speech emotion recognition with cross-lingual databases, is too coarse, and that the reference distribution should be neutral speech of the same language. Multi-Teacher Language-Aware Knowledge Distillation showed that monolingual acoustic teachers help a multilingual emotion model. KoCoSa showed in text that a Korean-pre-trained encoder beats zero-shot GPT-3.5 on sarcasm, 74.9% versus 53.5% balanced accuracy. iCompass showed that dialect-specific pre-training helps Arabic sarcasm detection. We carry this monolingual hypothesis to the acoustic expectation model and test it directly rather than assuming it.
Research Program
The pipeline has four parts.
Alignment. Each utterance is transcribed and force-aligned to phonemes and syllables. For English we use WhisperX with an English phoneme model. For Mandarin we use a Mandarin phoneme model plus pinyin with tone marks, so each syllable carries its lexical tone, and we run the Qwen3-ASR aligner as the multilingual alternative. Expressive and sarcastic speech stretches vowels and inserts odd pauses, which is exactly where aligners trained on read speech fail. We therefore measure boundary error against the manually segmented key phrases of the Functional Trade-off study for English, and against a small hand-corrected Mandarin subset we will annotate, before trusting anything downstream.
Expectation model. $f_L$ is a small transformer over the syllable sequence that regresses $p_k$. It is trained only on utterances labelled neutral in naturalistic podcast emotion corpora: MSP-Podcast for English and BIIC-Podcast for Taiwanese Mandarin. Its job is to learn what ordinary speech in that language sounds like given its words. We train three variants: monolingual per language, one pooled multilingual model that also includes Russian Dusha neutral speech, and a trivial lookup that predicts the mean prosody of each tone or stress class.
Detector. Token embeddings come from a language-specific text encoder: Chinese BERT, as used in MCSD, for Mandarin, and an English encoder for English. Each word token is concatenated with the pooled residuals of its syllables, passed through a two-layer transformer, and the classifier also receives the conversational context turns when the dataset provides them.
Analysis. Because each residual is attached to a word, we can ask which words carry the deviation. We test whether the residual magnitude peaks on the key phrases annotated in the Functional Trade-off study and whether its direction there matches that study's finding (lower $F_0$, longer, louder).
Baselines
Naive reference. A text-only classifier using the same language-specific encoder and context. MCSD reports 69.78% F1 for text alone and 74.94% for audio alone with an SVM (A Multimodal Chinese Dataset for Cross-lingual Sarcasm Detection). Any prosody method must clear text-only by a margin larger than seed variance to justify itself.
Published reference points. These are not run under our protocol and come from different split and modality settings, so they are context only. On MCSD, the published multimodal SVM reaches 76.64% macro-F1 (same paper). On MCSh, MMCSNN reports 86.17% macro-F1 with pooled OpenSMILE features plus BERT (Multimodal Chinese Sarcasm Detection Integrating Audio Attributes and Textual Features). On the balanced MUStARD++, a trimodal gated-attention system with context reaches 0.736 macro-F1 (Sarcasm in Sight and Sound); since it uses video, we rerun its audio and text branches only. On PodSarc, the bimodal model reports 73.63% F1 (Leveraging Large Language Models for Sarcastic Speech Annotation). For MSCD we found no reported macro-F1 on Lacuna, only the 40% false positive rate under manipulation. MCSD and MSCD appear to be the same 2,705-utterance stand-up corpus under two names, and we will verify this.
Strongest competitor. We reimplement two baselines on identical splits, with identical text encoders, five seeds and the same hyperparameter search budget of a fixed number of trials each:
- Unaligned cross-attention: text tokens attend to frame-level features from a sarcasm- or emotion-fine-tuned wav2vec 2.0. This is the best existing design and learns its own soft alignment.
- Aligned raw prosody: the CAN-style detector fed $p_k$ instead of $r_k$.
Zero-shot multimodal LLM. Qwen-Omni is run with the probing prompts of the Prosodic Heuristics study.
Ablations isolating the key ingredient. Holding the detector fixed, we vary the source of $r_k$: no expectation (raw), speaker z-scoring only, tone or stress class means, multilingual $f$, and monolingual $f_L$.
What counts as beating the baselines. Two things must both hold:
- Natural sarcasm: macro-F1 at least as high as the stronger reimplemented baseline, with overlapping or better confidence intervals.
- Manipulated sincere speech: a false positive rate on PSOLA-edited sincere utterances at most half that of the best baseline.
Winning on F1 alone would not support our claim.
Why not just…?
Why not just let a fine-tuned frame-level cross-attention model learn the alignment and the tone normalization itself? This is the most serious threat. Wav2vec 2.0 features fine-tuned on Chinese already give 74.94% audio-only F1 on MCSD. Sarcasm corpora are small, though: a few thousand utterances. Separating tone from attitude requires knowing which syllable carries which tone, and the Prosodic Heuristics study's discriminant analysis found false positives already tangled inside the audio encoder before text fusion. If the unaligned baseline matches us on both natural F1 and the manipulation false positive rate, the proposal's core claim is wrong and we will report it as such.
Why not just z-score pitch per speaker and pool over the utterance? Speaker normalization removes voice range but not lexical tone. The Functional Trade-off study found no utterance-level $F_0$ range differences even where key-phrase effects were significant. The speaker-only rung of the ablation ladder measures exactly how much this captures.
Why not just subtract the mean prosody of each tone class, with no learned predictor? This may well be enough, and we include it as an explicit rung. If it matches $f_L$, the contribution narrows to "lexically conditioned prosody via forced alignment" with a cheap expectation, which is still a usable result. The learned model can only justify itself through context effects that tone means miss, such as tone sandhi (tones changing next to other tones), downdrift (pitch falling across a phrase) and phrase-final lengthening.
Why not just prompt a multimodal LLM, or fine-tune it on sarcasm? Zero-shot, these models show the 40 to 60% false alarms documented by the Prosodic Heuristics study. A fine-tuned LLM could improve on natural data while keeping the heuristic, so we evaluate the zero-shot model under the same manipulation test rather than on F1 alone. EmotionThinker raised Qwen2.5-Omni from 50.83% to 68.89% on emotion benchmarks by explicitly rewarding prosodic reasoning, which suggests that explicit prosodic grounding is what such models lack.
Proposed Experiments and Decisive Checks
Main comparison. The full ablation ladder is run on MUStARD++ and PodSarc for English, and on MCSD/MSCD and MCSh for Mandarin, reporting macro-F1 over five seeds.
Manipulation stress test. We reproduce the PSOLA protocol of the Prosodic Heuristics study on the sincere test utterances of each language: pitch raised and pauses inserted. The metric is the false positive rate. As a control, we apply the "sarcastic" direction the Functional Trade-off study found for English (lowered $F_0$, lengthened key phrases) to the key words of sincere utterances. A grounded detector should respond more to this edit than to the generic expressive edit. A heuristic detector should show the reverse.
Tone confound. We compare the drop in Mandarin F1 when replacing $r_k$ with $p_k$ against the same drop in English. Claim two predicts a larger drop in Mandarin.
Monolingual expectation. We test whether monolingual $f_L$ beats multilingual $f$ on detection F1, on the manipulation false positive rate, and on held-out neutral prediction error. As a secondary check, we test whether residuals from $f_L$ reduce anger and happiness confusion in four-class emotion recognition on MSP-Podcast and BIIC-Podcast.
Cross-lingual transfer, exploratory. We train the residual-based detector head on English and apply it to Mandarin, compared with raw-prosody transfer. Better residual transfer would mean the residual coordinates are comparable across languages. A failure would be informative too, since English and Mandarin may encode attitude in opposite pitch directions.
Alignment robustness. Alignment error is a gating precondition. If boundary error on sarcastic key phrases is large enough to blur syllables together, we will first fine-tune the aligner on the hand-corrected subset, in the spirit of weakly-supervised alignment of disfluent speech. We will report results both with and without that fix.
References
- When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection
- A Functional Trade-off between Prosodic and Semantic Cues in Conveying Sarcasm
- Multimodal Chinese Sarcasm Detection Integrating Audio Attributes and Textual Features
- Multimodal Speech Emotion Recognition Using Cross Attention with Aligned Audio and Text
- WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
- Qwen3-ASR Technical Report
- Emotions in dubbed speech: an intercultural approach with respect to F0
- 台灣情緒語料庫建置與辨識 (An Emotional Speech Database in Taiwan: Collection and Recognition) [In Chinese]
- 以多層感知器辨識情緒於國台客語料庫 (Use Multilayer Perceptron To Recognize Emotion in Mandarin, Taiwanese and Hakka Database) [In Chinese]
- Analysis of Phonetic Level Similarities Across Languages in Emotional Speech
- Speech emotion recognition with cross-lingual databases
- Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition
- KoCoSa: Korean Context-aware Sarcasm Detection Dataset
- iCompass at Shared Task on Sarcasm and Sentiment Detection in Arabic
- A Multimodal Chinese Dataset for Cross-lingual Sarcasm Detection
- Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection
- Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection
- EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning
- Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling