Speech recognizers make mistakes. A popular fix is to hand a large language model the recognizer's top few guesses for an utterance (the "N-best list") and let it write a corrected transcript. This is called generative error correction. A natural next step is to also give the model the preceding turns of the conversation, so it can choose words that fit the dialogue. This proposal argues that, for sarcasm, this step can backfire. A sarcastic line is defined by not fitting its context: someone says "wow, absolute genius" right after a colleague deleted the database. A corrector trained to make words agree with the context is, by construction, pushed to "fix" exactly the clashing word that a sarcasm detector needs to see.
We propose to measure this effect, which we call incongruity erasure, on spoken sitcom sarcasm. We then test a simple remedy, incongruity-preserving correction. The corrector may use dialogue context only where the recognizer's own guesses disagree with one another. Where the guesses agree, it may not rewrite sentiment-bearing words. The study answers a practical question nobody has posed: does putting an LLM corrector in front of a context-aware sarcasm detector help, hurt, or do nothing? It also offers a concrete case where context conditioning in error correction has a measurable cost.
Core Claim
Context-conditioned generative correction lowers overall word error rate on sarcastic dialogue. It also raises the error rate on sentiment-bearing words in sarcastic utterances more than in sincere ones. As a result, it degrades downstream sarcasm detection relative to context-free correction. Restricting context-driven edits to positions where the N-best hypotheses diverge should recover the detection loss while keeping most of the transcription gain.
The claim is new because the pieces have only been studied apart. On one side, the literature shows that generative correction can go beyond the best single hypothesis in the N-best list, and that adding context helps. Generative Speech Recognition Error Correction with LLMs and Task-Activating Prompting fine-tunes LLMs on N-best lists and reaches 2.11% word error rate on WSJ, against a 10-best best-possible-selection error of 9.78%. Contextualized Speech Recognition adds a background Wikipedia passage and cuts SQuAD-SRC error from 24.2% to 13.1%.
On the other side, the sarcasm literature locates the signal in mismatch with context. Harnessing Context Incongruity for Sarcasm Detection formalizes sarcasm as a clash between an utterance's polarity and its situation. Sarcasm Analysis Using Conversation Context shows that the preceding turns carry this clash.
Put together, these two findings predict a conflict that neither line tested. Tung et al.'s context was factual reference text, where agreement with context is always the right answer. In conversation, disagreement is sometimes the message.
Research Program
Building the evaluation material. We start from MUStARD (Towards Multimodal Sarcasm Detection), which has 690 balanced sitcom clips with human transcripts and preceding dialogue turns. We add the MUStARD++ Balanced extension from Sarcasm in Sight and Sound, which contributes 164 House MD clips covering rarer sarcasm types. For every target utterance and every context turn, we run Whisper beam search to obtain 10-best lists. Context turns are therefore also recognizer output, as they would be in a deployed system.
The corrector needs more training data than these clips provide. We train it on conversational TV speech from MELD, the Friends-derived dialogue set underlying BMELD in Generative Error Correction for Emotion-aware Speech-to-text Translation. Because MUStARD also draws on Friends, we will remove every episode that appears in MUStARD to prevent leakage.
Correctors. All correctors share one 7B instruction-tuned model, fine-tuned with LoRA (low-rank adapters, a cheap way to fine-tune a large model), on identical data and an identical tuning budget. They differ only in what they see and what they may edit. The context-free corrector sees the 10-best list, following Yang et al. The context-conditioned corrector also sees the previous three turns. We choose three because A Report on the 2020 Sarcasm Detection Shared Task found gains plateau beyond that, which also keeps the prompt short.
The incongruity-preserving corrector sees the same input as the context-conditioned one, with one constraint on its output. We align the 10 hypotheses and mark each word position as "agreed" if most hypotheses share the same word there. This uses the voting signal described by Tan et al.. At agreed positions holding a sentiment-bearing word, the corrector may not change the word. We identify sentiment-bearing words with SentiWordNet, the lexicon Sarcasm SIGN used to locate the words that carry irony. The constraint is enforced at decoding time by forcing the agreed word.
A second variant enforces the constraint during training instead. Training targets are left unchanged, but the corrector is penalized for edits at agreed positions. This tests whether the corrector can learn the behavior rather than having it imposed.
Measurements. We report overall word error rate. We also report a polarity-word error rate, which is word error rate restricted to SentiWordNet words in the reference transcript, split by sarcastic and sincere labels.
The main diagnostic is a counterfactual context swap. For each sarcastic target, we replace its true preceding turns with turns drawn from a sincere clip, so the target no longer clashes. We then measure how often the corrector's output for polarity words changes. A corrector that is reading the acoustics should not change these words when the context changes. A corrector that is enforcing fit with the context will change them.
Downstream, the corrected transcripts are fed to a fixed context-aware sarcasm detector. Text-only and trimodal versions both follow Bhosale et al.'s gated-attention design. We use speaker-independent five-fold splits and report macro-F1 with bootstrap confidence intervals. Given roughly 850 clips, we treat differences smaller than the interval width as null.
Baselines
The reference points are the clean-transcript detector and raw Whisper 1-best output.
The clean-transcript detector is the upper bound. Bhosale et al. report macro-F1 of 0.736 on MUStARD++ Balanced for their trimodal model with context, using human transcripts. Castro et al. report 63.1% F-score for the speaker-independent text-plus-audio model on MUStARD. We retrain the same detector ourselves, so all comparisons use our own split and seeds rather than these published figures. No result was found on Lacuna for sarcasm detection on recognizer-generated transcripts. The size of the gap between clean and recognized transcripts is therefore itself a first finding.
The do-nothing baseline feeds Whisper's single best hypothesis to the same detector, with no correction.
The strongest existing correction method is context-free N-best correction in the style of Yang et al. Their reported best is 2.11% WSJ error from a LoRA-tuned OpenLLaMA. This baseline is the strongest competitor because it already goes beyond what selecting from the list can achieve, and it cannot erase incongruity through context because it never sees context.
The second competitor is context-conditioned correction, our adaptation of Tung et al. from Wikipedia passages to dialogue turns. It is expected to win on overall word error rate, and the hypothesis predicts it loses on sarcasm F1.
The key ablation is the incongruity-preserving corrector with its constraint randomized. It freezes the same number of words, but at randomly chosen positions rather than agreed sentiment-bearing ones. This separates the effect of where we block edits from the effect of simply editing less, which is how generic over-correction regularization works, for example the continue-training used in Crossmodal ASR Error Correction with Discrete Speech Units.
All correctors share base model, data, LoRA rank, and number of training steps. The proposal counts as beating the baselines if three things hold. First, incongruity-preserving correction matches or exceeds context-free correction on sarcasm macro-F1 with non-overlapping confidence intervals against context-conditioned correction. Second, it keeps at least half of context-conditioned correction's word error rate reduction. Third, it beats the random-constraint ablation.
Why not just…?
Why not just feed Whisper's 1-best transcript to the detector? This is the most serious threat. Whisper may transcribe sitcom dialogue well enough that the few remaining errors rarely touch the words that matter. The Crossmodal ASR Error Correction paper found that correction improved downstream emotion recognition only from 60.92% to 61.82% unweighted accuracy, a small effect. The clean-versus-1-best experiment settles this directly. If the detection gap is within noise, we will say that correction is unnecessary for this task. The erasure measurement would still stand as a finding about how context-conditioned correctors behave, independent of whether correction is worth deploying.
Why not just give the whole N-best list to the sarcasm detector? This sidesteps correction entirely and lets the detector weigh the alternatives itself. It is a plausible competitor, so we include it as a baseline with the same detector backbone. Its weakness is that it cannot produce a word that appears in no hypothesis. Yang et al.'s gap between the 9.78% best-possible-selection error and the 2.11% corrected error shows how often such words matter in other domains. Whether that holds for sitcom speech is an open empirical question.
Why not just use an end-to-end audio language model to classify sarcasm directly? Commander-GPT reports that LLMs struggle with sarcasm without structured reasoning. An end-to-end model also gives no transcript to inspect, so erasure cannot be diagnosed. We will run one such model zero-shot as a reference point. We do not claim our pipeline beats it unless the numbers show that.
Why not just regularize the corrector to edit less? Generic conservatism is exactly what the random-constraint ablation tests. If freezing random words performs as well as freezing agreed sentiment-bearing words, then our targeting idea adds nothing, and we will report that.
Decisive Checks
The hypothesis lives or dies on two numbers. The first is the sarcastic-minus-sincere difference in polarity-word error rate under context-conditioned correction, compared with the same difference under context-free correction. The second is the polarity-word flip rate under the counterfactual context swap.
If context-conditioned correction does not raise polarity-word errors specifically on sarcastic clips, there is no erasure effect. The remedy is then pointless, and the honest conclusion is that dialogue context is safe to use in correction. If erasure does appear, the sarcasm F1 comparison against the clean-transcript upper bound shows how much of the damage the constrained corrector repairs.
References
- Generative Speech Recognition Error Correction With Large Language Models and Task-Activating Prompting
- Contextualized Speech Recognition: Rethinking Second-Pass Rescoring with Generative Large Language Models
- Harnessing Context Incongruity for Sarcasm Detection
- Sarcasm Analysis Using Conversation Context
- Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
- Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection
- Generative Error Correction for Emotion-aware Speech-to-text Translation
- A Report on the 2020 Sarcasm Detection Shared Task
- FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition
- Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation
- Crossmodal ASR Error Correction With Discrete Speech Units
- Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detection