Integration of Dubbing Constraints into Machine Translation

Ashutosh Saboo, Timo BaumannACL 2019

TL;DR

This paper tackles the challenge of making machine translation (MT) systems produce "dubbable" output, meaning translations that not only preserve meaning but also align with lip movements and phonetic synchrony. They introduce a method to integrate these dubbing constraints into neural MT decoders, showing it can significantly improve dubbability with only a minor impact on translation quality metrics like BLEU.

What Problem Does This Paper Address?

Dubbing, the process of revoicing foreign media to match lip movements, is crucial for accessibility and immersion. Traditional machine translation focuses on meaning preservation, often resulting in translations that are too long or have different phonetic characteristics, making them impossible to dub lip-synchronously. Existing speech-to-speech translation systems don't inherently address these dubbing constraints. The authors aim to bridge this gap by developing MT systems that can control the trade-off between translation faithfulness and "dubbability," enabling more natural and accessible dubbed content.

Methods

The core of their approach involves integrating phonetic synchrony constraints into the MT decoding process. They propose a "dubbing estimator" that works within the MT decoder's search mechanism. This estimator calculates a combined score for candidate translations, factoring in both the standard translation faithfulness score and a phonetic synchrony score.

Specifically, they use syllable count as a proxy for phonetic synchrony, assuming that a similar number of syllables in the source and target languages will facilitate lip-synchronous revoicing. Their method involves:

  • N-best Rescoring: They first generate an N-best list of translations using a standard MT decoder.
  • Syllable Difference Calculation: For each candidate translation and the source sentence, they calculate the absolute difference in syllable counts.
  • Synchrony Score: This difference is converted into a synchrony score, where identical syllable counts yield the highest score.
  • Weighted Combination: The translation faithfulness score and the synchrony score are combined using a weight ($\alpha$) that balances their relative importance. This allows for tuning the system to prioritize dubbability over pure meaning preservation, or vice-versa.

They also discuss a more integrated approach where synchrony constraints are considered during the decoding process, estimating the remaining phonetic score based on the decoder's internal state. However, their initial prototype focuses on the rescoring approach for simplicity.

Datasets

  • HEROes dubbing corpus: This corpus, containing aligned English source sentences and Spanish dubbed revoicings from the TV show "Heroes," is used for evaluation. It's described as the largest available corpus of dubbed speech.
  • Europarl corpus: This corpus is used for training the baseline MT model due to the limited size of the HEROes corpus for training. A test set is also derived from Europarl for in-domain evaluation.

Key Insights and Ideas

  • Syllables as a Proxy for Dubbability: The paper posits that syllable count is a reasonable, albeit simplified, proxy for the phonetic and visemic synchrony required for dubbing. This is a practical simplification that makes the problem tractable.
  • Trade-off Management: The introduction of a tunable weight ($\alpha$) is key. It allows for explicit control over the balance between translation quality and dubbability, acknowledging that perfect adherence to both is often impossible.
  • Integrated vs. Rescoring Approaches: The paper highlights two ways to integrate constraints: a more complex, integrated decoder approach and a simpler, post-hoc N-best list rescoring. The latter is easier to implement and provides a good baseline.
  • Domain Mismatch Impact: The evaluation on the Europarl corpus (out-of-domain for the HEROes-trained model) demonstrates that domain mismatch significantly impacts BLEU scores, but the synchrony improvements still hold, suggesting the dubbing constraints are somewhat robust.

Key Results

  • Improved Dubbability: The proposed method significantly improves the "dubbability" of translations, as measured by a synchrony score (lower is better). For $\alpha = 0.3$, the syllable difference between source and target is halved from nearly a quarter to 14%.
  • Minimal BLEU Degradation: The impact on BLEU scores is relatively small, especially for lower values of $\alpha$. For $\alpha = 0.3$, BLEU score decreased by only about 2%.
  • Steeper Improvement in Dubbability: The paper notes that dubbability improves more steeply than BLEU degrades, suggesting that even small adjustments can yield substantial gains in lip-synchrony.
  • Beam Size Importance: For the rescoring approach, a larger beam size (e.g., 50) is found to be essential for providing sufficient candidates for effective rescoring, without significantly degrading BLEU.

Limitations and Future Work

  • Syllable Count as a Simplification: The authors acknowledge that syllable count is a proxy and not a perfect measure of phonetic synchrony. More sophisticated constraints based on accentuation, stress, duration, articulatory features, and visemes are needed for human-level performance.
  • Limited Training Data: The HEROes corpus, while the largest available for dubbing, is still too small to train an NMT model from scratch. This necessitates using larger, out-of-domain corpora like Europarl for training.
  • Domain Mismatch: The performance on out-of-domain data (Europarl) shows a lower BLEU score, indicating that domain adaptation techniques might be necessary.
  • Future Work:
    • Implementing the fully integrated search approach within the decoder.
  • Developing more powerful synchrony metrics that consider source audio (stress, articulation) and video (face visibility).
  • Incorporating target speech synthesis cost estimates for viseme alignment.
  • Exploring models that can generate alternative wordings for the same concept with different synchrony properties.

Summary updated

BibTeX
@inproceedings{saboo2019integration,
  author = {Ashutosh Saboo and Timo Baumann},
  title = {{Integration of Dubbing Constraints into Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure