Using Deepfake Technologies for Word Emphasis Detection

Eran Kaufman, Lee-Ad Gottlieb

Abstract: In this work, we consider the task of automated emphasis detection for spoken language. This problem is challenging in that emphasis is affected by the particularities of speech of the subject, for example the subject accent, dialect or voice. To address this task, we propose to utilize deep fake technology to produce an emphasis devoid speech for this speaker. This requires extracting the text of the spoken voice, and then using a voice sample from the same speaker to produce emphasis devoid speech for this task. By comparing the generated speech with the spoken voice, we are able to isolate patterns of emphasis which are relatively easy to detect.

Introduction

Prosody—the rhythm, stress, and intonation of speech—is a primary carrier of meaning in human communication. Detecting which word a speaker is emphasizing is notoriously difficult for machines because "normal" speech varies wildly across different accents and dialects. A shout for one person might be the baseline volume for another. To solve this "moving target" problem, this paper proposes a novel approach: using deepfake technology to generate a personalized, neutral baseline for every speaker.

Methodology

The authors leverage the SV2TTS (Speaker Verification to Multispeaker Text-to-Speech) architecture to create a personalized "vanilla" reference. By generating a synthetic version of the speaker's own voice, the system can compare the original audio to a version that is naturally devoid of emotional nuance.

Figure 1: The proposed system architecture, showing the pipeline from speech waveform to word emphasis detection.

As shown in the architecture above, the pipeline consists of several key stages:

  1. Encoder: Processes a sample of the subject's speech to create a unique "voice print" (embedding vector).
  2. Speech-to-Text (STT): Transcribes the query sentence into raw text.
  3. Synthesis: Uses the raw text and the voice print to synthesize a new audio file. Standard neural TTS models tend to produce flat, emotionless speech, making them perfect "control groups."
  4. Word Separation: Both the original and synthetic audio are segmented into individual words using a sliding Root Mean Square (RMS) window.
  5. Waveform Comparison: The system compares the original word to the synthetic word using Fast Fourier Transforms (FFT) and cross-correlation.
Figure 4: Visualization of word separation using a sliding RMS window for the sentence "I did not take your bag."

Experimental Setup

The underlying TTS and vocoder (WaveNet) were trained on the VCTK dataset, containing 44 hours of speech from 109 speakers. For evaluation, the researchers created a custom dataset of 100 voice samples where five different speakers recited five sentences, with emphasis placed on one of four different words in each sentence.

Key Results and Insights

The system proved highly effective at isolating emphasized words by comparing the "real" vs. "vanilla" versions of the same voice. This approach effectively eliminates the "accent gap" by comparing the speaker to a neutral version of themselves rather than a universal average. The detector specifically looks for shifts in Pitch (central frequency) and Skew (spectral distribution spread).

The quantitative performance metrics were impressive:

  • Accuracy: 92%
  • Precision: 89.14%
  • Recall: 89.33%
  • F1 Score: 89.23%

Conclusion and Future Work

This research demonstrates that the inherent lack of emotion in standard AI synthesis—often seen as a limitation—can be repurposed as a powerful scientific tool. By measuring the "distance" between a user's actual speech and their synthesized neutral baseline, machines can more accurately interpret prosodic meaning. Future work will aim to expand this technique to classify complex emotional states like sarcasm or frustration, potentially revolutionizing sentiment analysis in voice-activated assistants.

Summary updated

Abstract

In this work, we consider the task of automated emphasis detection for spoken language. This problem is challenging in that emphasis is affected by the particularities of speech of the subject, for example the subject accent, dialect or voice. To address this task, we propose to utilize deep fake technology to produce an emphasis devoid speech for this speaker. This requires extracting the text of the spoken voice, and then using a voice sample from the same speaker to produce emphasis devoid speech for this task. By comparing the generated speech with the spoken voice, we are able to isolate patterns of emphasis which are relatively easy to detect.
BibTeX
@inproceedings{kaufman2023using,
  author = {Eran Kaufman and Lee-Ad Gottlieb},
  title = {{Using Deepfake Technologies for Word Emphasis Detection}},
  booktitle = {PACLIC},
  year = {2023},
  url = {https://openreview.net/forum?id=XWa3YIvah1},
}
Figure