Abstract
Sarcastic speech synthesis, which involves generating speech that effectively conveys sarcasm, is essential for enhancing natural interactions in applications such as entertainment and human-computer interaction. However, synthesizing sarcastic speech remains a challenge due to the nuanced prosody that characterizes sarcasm, as well as the limited availability of annotated sarcastic speech data. To address these challenges, this study introduces a novel approach that integrates feedback loss from a bi-modal sarcasm detection model into the TTS training process, enhancing the model's ability to capture and convey sarcasm. In addition, by leveraging transfer learning, a speech synthesis model pre-trained on read speech undergoes a two-stage fine-tuning process. First, it is fine-tuned on a diverse dataset encompassing various speech styles, including sarcastic speech. In the second stage, the model is further refined using a dataset focused specifically on sarcastic speech, enhancing its ability to generate sarcasm-aware speech. Objective and subjective evaluations demonstrate that our proposed methods improve the quality, naturalness, and sarcasm-awareness of synthesized speech.
Introduction
Synthesizing sarcasm represents a "boss level" challenge for modern Text-to-Speech (TTS) systems. Unlike standard "read" speech, sarcasm relies on complex, non-literal prosodic cues, including exaggerated intonation, specific pitch shifts, and unexpected pauses. Current systems struggle primarily due to two factors: the scarcity of high-quality annotated sarcastic datasets and the inherent difficulty in capturing the subtle acoustic markers that distinguish a sincere statement from a sarcastic one.
Methodology
The proposed framework extends the FastSpeech 2 architecture with two primary innovations designed to bridge the gap between literal and sarcastic delivery.
Bi-Modal Sarcasm Detector and Feedback Loss
The core innovation is the integration of a pre-trained, frozen bi-modal sarcasm detector into the TTS training loop. This detector processes both text (via a BERT encoder) and audio (mel-spectrograms) to identify sarcastic intent.
During training, the TTS model generates a mel-spectrogram which is then fed into the detector. The system calculates a feedback loss based on the cosine similarity between the sarcasm embeddings of the ground-truth audio and the synthesized audio. This mechanism penalizes the TTS model if its output does not "cluster" with real sarcastic speech in the detector's latent space, effectively coaching the model to learn the specific acoustic markers of sarcasm.
Two-Stage Fine-Tuning
To overcome data scarcity and transition from robotic to natural conversational speech, a progressive training strategy was employed:
- Pre-training: Initial training on the LibriTTS dataset (clean, read speech).
- Stage I (Conversational): Fine-tuning on a 6.17-hour dataset of "in-the-wild" conversational speech from sitcoms (Friends, The Big Bang Theory) to learn natural dialogue rhythms.
- Stage II (Sarcastic): Final refinement on the MUStARD++ dataset, which contains 1,202 labeled sarcastic utterances, to hone the specific delivery of snark.
Key Results
The proposed model was evaluated against a standard FastSpeech 2 baseline using both objective metrics and subjective human listening tests.
Objective Evaluation
A sarcasm detector was used to evaluate the synthesized speech. The proposed model achieved an F1-score of 70.1% in sarcasm detection on its generated samples, a significant improvement over the 62.2% achieved by the baseline model. This indicates that the feedback loss successfully encouraged the generation of recognizable sarcastic markers.
Subjective Evaluation
Human listeners consistently favored the proposed model. In preference tests, 53% of participants rated the proposed model as sounding "more sarcastic" than the baseline. Furthermore, the proposed model achieved a higher Mean Opinion Score (MOS) for sarcasm awareness. Notably, 15% of listeners gave the proposed model the highest possible score (5) for sarcasm awareness, triple the 5% awarded to the baseline.
Discussion and Future Work
The study highlights that sarcasm detection—and by extension, synthesis—is significantly more accurate when incorporating both text and audio. This suggests that the model benefits from understanding "semantic incongruity"—the gap between literal words and intended meaning.
While the results are promising, several areas for future research remain. Sarcasm is not binary; modeling varying levels of "snark" would allow for more controllable and nuanced synthesis. Additionally, future tests should include neutral control sentences to ensure the model can distinguish when not to be sarcastic, avoiding potential priming effects in listeners.
Conclusion
By integrating a bi-modal sarcasm detector as a "critic" and employing a two-stage fine-tuning process, this research demonstrates a viable path toward generating more natural and expressive sarcastic speech. The use of feedback loss moves beyond simple spectrogram matching toward optimizing for the perceptual category of sarcasm.
Summary updated