Using Deepfake Technologies for Word Emphasis Detection

Eran Kaufman, Lee-Ad Gottlieb, Dina Mayzlish, Or Tiram, Hila Wiesel, Nofar Yosef

Introduction

In spoken language, prosody—the rhythm, stress, and intonation of speech—is a primary carrier of meaning. Consider the sentence: "I did not take your bag." Depending on which word is emphasized, the meaning shifts from "Someone else took it" to "I took a different bag." Detecting these nuances automatically is notoriously difficult because every voice has a unique baseline pitch and rhythm. This paper proposes a novel solution: using "deepfake" technology to generate a perfectly neutral, emphasis-devoid version of a speaker’s own voice to serve as a scientific control.

The Speaker-Specific Challenge

Traditional methods for emphasis detection rely on "hand-picked" features like fundamental frequency ($F_0$), energy, and duration. However, these features vary wildly based on a speaker's accent, dialect, and natural pitch. A high-pitched word for a bass singer might be a low-pitched word for a soprano. The authors tackle this hurdle by moving away from absolute measurements and toward a comparative approach, using generative AI to create a personalized baseline.

Proposed Methodology

The authors introduce a pipeline that uses generative speech models to create a personalized baseline for comparison. The workflow consists of five main stages, visualized in the architecture diagram below:

The proposed pipeline for word emphasis detection using a synthesized neutral baseline.
  1. Voice Encoder: Processes a pre-recorded sample of the user’s voice to create a representative "embedding vector"—a digital fingerprint of their vocal characteristics.
  2. Speech-to-Text (STT): Extracts the literal text from the "query" sentence (the audio to be analyzed).
  3. Text-to-Speech (TTS) & Vocoder: Using the embedding vector and the extracted text, the system generates a new audio waveform. Because standard TTS models are "vanilla" (struggling to replicate emotional nuance), the resulting "deepfake" is a neutral version of the speaker reciting the same text.
  4. Word Separation: Both the original and the deepfake audio are sliced into individual words using a sliding Root Mean Square (RMS) window to identify silences.
Word separation using Root Mean Square (RMS) energy levels to identify silences between words in the sentence "I did not take your bag."
  1. Waveform Comparison: The system performs a cross-correlation of the spectrograms for each word. If a word was spoken naturally (without emphasis), it correlates highly with the deepfake version. If it was emphasized, the correlation is significantly lower.

Key Insights and Results

The researchers identified two distinct ways emphasis ruins correlation: Pitch Accents, where the speaker shifts the frequency ($F_0$), and Continuous Modulation, where the speaker varies their voice dynamically during the word. By using cross-correlation instead of absolute feature extraction, the system becomes robust across different languages and dialects.

Performance Metrics

The system performed remarkably well on a custom evaluation dataset of over 100 voice samples from five different speakers:

  • Accuracy: 92%
  • Precision: 89.14%
  • Recall: 89.33%
  • F1 Score: 89.23%

Qualitatively, for non-emphasized words, the cross-correlation peak is close to zero (indicating high similarity). For emphasized words, frequency shifts of approximately 80Hz (roughly 30% of the fundamental frequency) were observed, which the system easily flagged.

Conclusion and Future Work

While deepfakes are often viewed with suspicion, they provide a powerful tool for understanding the subtle "music" of human speech. Future work involves moving beyond "recitation speech" to spontaneous conversation and integrating linguistic context—such as knowing that certain word orders typically imply emphasis—to further refine detection accuracy. Potential downstream applications include automated translation that preserves meaning and more nuanced sentiment analysis.

Summary updated

Abstract

In this work, we consider the task of automated emphasis detection for spoken language. This problem is challenging in that emphasis is affected by the particularities of speech of the subject, for example the subject accent, dialect or voice. To address this task, we propose to utilize deep fake technology to produce an emphasis devoid speech for this speaker. This requires extracting the text of the spoken voice, and then using a voice sample from the same speaker to produce emphasis devoid speech for this task. By comparing the generated speech with the spoken voice, we are able to isolate patterns of emphasis which are relatively easy to detect.
BibTeX
@inproceedings{kaufman2023using,
  author = {Eran Kaufman and Lee-Ad Gottlieb and Dina Mayzlish and Or Tiram and Hila Wiesel and Nofar Yosef},
  title = {{Using Deepfake Technologies for Word Emphasis Detection}},
  booktitle = {PACLIC},
  year = {2023},
  url = {https://openreview.net/forum?id=XWa3YIvah1},
}
Figure