Prosodic Phrase Alignment for Machine Dubbing

Alp Öktem, Mireia Farrús, Antonio BonafonteInterspeech 2019

TL;DR

This paper tackles the challenge of machine dubbing by proposing a novel method to align prosodic phrases between source and translated audio. By leveraging attention mechanisms from neural machine translation, they can transfer the original phrasing structure to synthesized speech, leading to more natural-sounding dubbed content and improved lip-syncing.

What Problem Does This Paper Address?

Audiovisual translation, particularly dubbing, aims to replace original dialogue with translated speech that appears to be naturally spoken by the on-screen actors. This requires not only linguistic accuracy but also a faithful reproduction of the original's prosody – the rhythm, intonation, and phrasing. Achieving this synchronization, especially for lip-syncing, is crucial for visual coherence and audience immersion. While subtitling is more automated, dubbing presents a greater challenge due to the need for spoken output that matches visual cues. Existing automated dubbing solutions often fall short by treating subtitle segments as independent units, leading to unnatural prosody, or by not adequately addressing the nuanced alignment of spoken phrases with original timing and rhythm. This paper addresses the gap in automated dubbing by focusing specifically on aligning prosodic phrases between source and target languages to improve the naturalness and synchronization of synthesized dubbed audio.

Methods

The core of the proposed methodology lies in two key stages: cross-lingual phrase alignment and synthesis timing synchronization.

  1. Cross-lingual Phrase Alignment using Attention: The authors exploit the attention weights generated by a Neural Machine Translation (NMT) system. These weights, typically used to help the model focus on relevant parts of the source sentence during translation, are repurposed to infer alignment between source and target prosodic phrases. They define prosodic phrases as voiced segments followed by a silent pause (longer than 250ms). The attention mechanism's output is used to identify potential target phrase structures that mirror the source's prosodic phrasing. This involves a two-stage process: generating possible target phrase label sequences and then scoring them based on the attention weights to find the most appropriate alignment.
  2. Synthesis Timing Synchronization: Once the target prosodic phrases are aligned, their durations are conditioned to match the source. This is achieved by directly mapping the durational parameters of source prosodic phrases to their aligned target counterparts. A "bending ratio" is calculated for each phrase, representing the desired duration versus the TTS system's predicted duration. This ratio is then used to adjust the phoneme and pause durations within the target phrase before synthesis, ensuring the synthesized speech respects the original timing and phrasing.

Datasets

  • Heroes Corpus: This is the primary dataset used for both training and evaluation. It's a collection of parallel English and Spanish dialogue segments from a North American TV series, complete with transcripts and prosodic annotations (including word-level alignment and silent pause durations). It contains 7000 parallel segments.
  • WIT-TED Corpus: Used for initial training of the machine translation models before fine-tuning on the Heroes Corpus.

Key Insights and Ideas

  • Attention as a Prosodic Alignment Tool: The paper demonstrates that the attention mechanism in NMT, often seen as a byproduct for translation quality, can be effectively repurposed to infer cross-lingual prosodic phrase alignment. This is a novel application of a standard NMT component.
  • Prosodic Phrase Structure Transfer: The core idea is that by aligning prosodic phrases, the system can transfer the "natural rhythm" of the original speech to the translated and synthesized output, rather than just aligning word-for-word or sentence-by-sentence.
  • Duration Conditioning for Naturalness: Directly conditioning the synthesis timing based on source phrase durations, rather than just relying on subtitle timestamps, is crucial for achieving more natural-sounding dubbed speech that aligns with lip movements.
  • Pauses as Key Indicators: The study highlights the importance of pauses (specifically those longer than 0.1-0.4 seconds) as strong indicators of prosodic phrase boundaries that are often preserved in professional dubbing.

Key Results

  • Speech Rate Ratio: The average speech rate ratio between source and automatically aligned target segments was found to be 1.27, which is close to the 1.31 ratio observed in professionally dubbed segments of the Heroes Corpus.
  • Pause Reflection: For pauses longer than 0.1 seconds, there was a 70% likelihood of reflection in dubbing translation. This increased to 81% for 0.2-second pauses and 91% for 0.4-second pauses, indicating the importance of long pauses in professional dubbing.
  • Perception Test (MOS):
    • Translation Quality: The proposed "synced" system scored 3.44, while a "subtitle" baseline scored 4.08. This suggests that while the translation quality itself might be slightly lower than a direct subtitle approach (which might prioritize literal translation), the overall experience is still good.
  • Lip-syncing Precision: The "synced" system achieved a score of 3.58, compared to 2.96 for the "subtitle" baseline, indicating a significant improvement in lip-syncing quality.
  • Preference: Participants preferred the "synced" system 68% of the time over the "subtitle" system, strongly indicating the perceived improvement in dubbing quality.

Limitations and Future Work

  • Translation Quality Bottleneck: The authors acknowledge that poor translation quality itself can negatively impact the perceived dubbing quality, even if the prosody and lip-sync are good. Test participants preferred professional translations even if their lip-syncing was less precise.
  • Unnatural Synthesis: Bending ratios that are too high or too low can lead to unnatural synthesized speech. Further work is needed to integrate modalities that achieve better length matching in terms of both syllable count and articulation duration.
  • Limited Lip-syncing Guidelines: The current system aligns syntheses to spoken intervals. Future work should incorporate more detailed lip-syncing guidelines, such as aligning open and closed phonemes, to achieve even more realistic synchronicity.
  • Future Directions:
    • Develop a scoring mechanism for phoneme-level alignment.
  • Exploit N-best translation results to potentially find better alternative translations that might improve speech rate ratios.
  • Integrate more sophisticated lip-syncing guidelines.

Summary updated

BibTeX
@inproceedings{oktem2019prosodic,
  author = {Alp Öktem and Mireia Farrús and Antonio Bonafonte},
  title = {{Prosodic Phrase Alignment for Machine Dubbing}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure