Acoustic Pairing of Original and Dubbed Voices in the Context of Video Game Localization

Adrien Gresse, Mickael Rouvier, Richard Dufour, Vincent Labatut, Jean-François BonastreInterspeech 2017

TL;DR

This paper tackles the challenge of automatically matching original voice actors with suitable dubbed voices for video game characters. They propose a novel approach using i-vectors and a learned projection matrix to bridge the language gap, significantly improving the accuracy of identifying acoustically similar voice segments across English and French.

What Problem Does This Paper Address?

Voice casting for video game localization is a complex and subjective process. Traditionally, human operators select voices, but this can be inconsistent and limited by the operator's personal biases and familiarity with certain actors. The goal is to automate this process by finding dubbed voices that acoustically match the original character's voice, preserving the character's persona and the player's immersion. This is particularly challenging because the dubbed voice needs to capture the character's essence, not just be a generic voice in the target language. The paper specifically addresses the problem of comparing voice segments across different languages, which introduces significant acoustic variability.

Methods

The core of the proposed approach is an extension of speaker recognition techniques, specifically using i-vectors and Probabilistic Linear Discriminant Analysis (PLDA).

  • i-vectors: These are low-dimensional representations that capture speaker characteristics from audio segments. The authors use language-dependent i-vectors, meaning separate models are trained for English and French.
  • PLDA: This statistical model is used to estimate the similarity between i-vectors, providing a likelihood ratio score.
  • Language Adaptation (Key Novelty): To overcome the language barrier, the authors introduce a transformation matrix (W). This matrix is learned to project i-vectors from one language's space to another. This allows for a more direct comparison of original (source) and dubbed (target) voice segments, even when they are in different languages. They explore three system configurations:
  • System A (Baseline): Compares i-vectors extracted using language-specific models without any cross-lingual projection.
  • System B: Projects source language i-vectors to the target language space using the learned matrix W.
  • System C: Projects target language i-vectors to the source language space using the learned matrix W.

Datasets

The authors created a custom corpus for their experiments by extracting dialogs from the video game "Mass Effect 3."

  • Corpus: Consists of audio segments from both the English (original) and French (dubbed) versions of the game.
  • Content: Includes 10,000 audio segments per language, featuring 38 characters voiced by at least 30 different actors.
  • Splits: The corpus was split into 70% for training, 10% for development, and 20% for testing.
  • Training Data for Models: For speaker recognition systems, they utilized standard datasets like NIST SRE 2004-2006 for English and ESTER-1, ESTER-2, EPAC, ETAPE, and REPERE for French.

Key Insights and Ideas

  • Bridging the Language Gap is Crucial: Simply comparing i-vectors from different languages (System A's EN-FR configuration) without adaptation yields poor results, highlighting the significant acoustic differences introduced by language.
  • Learned Projection Improves Similarity Detection: The learned projection matrix (W) in Systems B and C significantly boosts the accuracy of identifying acoustically similar voice segments across languages. This suggests that the projection effectively learns a "speaker mapping" between languages.
  • Character vs. Speaker Ambiguity: The results indicate that while the system can identify similar voices, there's an inherent ambiguity between distinguishing a speaker's voice and a character's persona. This is a fundamental challenge in voice casting.
  • Stereotypes Matter: The paper acknowledges that voice casting often relies on stereotypes (e.g., a soldier's voice). While not directly modeled, the system's ability to capture acoustic similarities indirectly supports preserving these perceived traits.

Key Results

  • Baseline (System A): Achieved accuracies of around 60.7-60.9% for same-language comparisons (EN-EN, FR-FR). The cross-lingual EN-FR comparison without adaptation was significantly lower (53.9%).
  • Adapted Systems (B & C): Showed substantial improvement. The best results were achieved with the EN-EN configuration, reaching 71.7% (System B) and 70.7% (System C) accuracy for the 3-best hypothesis.
  • Cross-lingual Performance: Systems B and C also performed well in the EN-FR configuration (around 68.9-69.6%), demonstrating the effectiveness of the projection.
  • Confusion Matrix Analysis: A confusion matrix for System B (EN-EN, k=1) revealed that the system performed better on more represented characters. It also highlighted that errors can arise when a single actor voices multiple characters, or when a character has distinct voice variations (e.g., young vs. old).

Limitations and Future Work

  • Ambiguity between Speaker and Character: The authors acknowledge that their current approach struggles to fully disentangle the speaker's identity from the character's persona. This is a key limitation for true character-based voice casting.
  • Corpus Limitations: The training data used for the i-vector models (NIST SRE, ESTER, etc.) might not perfectly represent the acoustic characteristics of acted voices in video games.
  • Future Directions:
    • Improved Acoustic Features: Explore more sophisticated acoustic features beyond MFCCs for better character representation.
  • Contrastive Experiments: Design experiments to specifically address the speaker-character ambiguity.
  • Character Classes: Investigate using character archetypes or classes to reduce variability and improve generalization.
  • More Diverse Data: Utilize more representative datasets for training and evaluation.

Summary updated

BibTeX
@inproceedings{gresse2017acoustic,
  author = {Adrien Gresse and Mickael Rouvier and Richard Dufour and Vincent Labatut and Jean-François Bonastre},
  title = {{Acoustic Pairing of Original and Dubbed Voices in the Context of Video Game Localization}},
  booktitle = {Interspeech},
  year = {2017},
}
Figure