Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems

Daniel Platnick, Bishoy Abdelnour, Eamon Earl, Rahul Kumar, Zahra Rezaei, Thomas Tsangaris, Faraj LagumACL 2024

TL;DR

This paper introduces Preset-Voice Matching (PVM), a novel framework for Speech-to-Speech Translation (S2ST) that prioritizes privacy by matching input voices to pre-consented speaker voices instead of cloning them directly. This approach not only enhances regulatory compliance and mitigates misuse risks but also demonstrably improves S2ST system efficiency and naturalness in multilingual scenarios.

What Problem Does This Paper Address?

The rapid advancement of AI-driven voice cloning technology has enabled powerful Speech-to-Speech Translation (S2ST) systems, particularly for applications like automated video dubbing. However, these systems often rely on cloning arbitrary input voices, raising significant privacy concerns. This lack of regulation can lead to misuse, infringement of personality rights, and the spread of misinformation through deepfakes. Existing S2ST systems, while effective, are vulnerable to these issues, and there's a pressing need for frameworks that integrate privacy safeguards without sacrificing performance.

Methods

The core innovation is the Preset-Voice Matching (PVM) framework. Instead of directly cloning the input speaker's voice, PVM operates in three stages:

  1. Similarity Feature Extraction: This module analyzes the input voice to extract meaningful features, such as gender and emotion.
  2. Target-language Preset-Voice Libraries: This acts as a curated database of pre-consented speaker voices in various target languages. The extracted features from the input voice are used to find the most similar consenting preset-voice in this library.
  3. Text-to-Speech with Matched Preset-Voice: A standard TTS module then synthesizes the translated text using the identified preset-voice, effectively creating a clone of the preset-voice that matches the characteristics of the input voice, rather than the input voice itself.

The paper also introduces GEMO-Match, a specific implementation of PVM that uses a hierarchical gender-emotion classifier for feature extraction. This classifier is trained on a newly created dataset.

Datasets

  • RAVDESS: A benchmark emotional speech dataset used for training and evaluating the gender-emotion classifiers. The authors filtered it to include only "strong" intensity samples for 5 common emotions (happy, angry, sad, disgust, calm).
  • Combined Gender-Dependent Dataset (CGDD): A newly curated dataset created by combining four benchmark emotional speech datasets (RAVDESS, CREMA-D, SAVEE, and TESS). CGDD aims to provide greater variance in speaker demographics and emotional intensities, facilitating more robust gender-dependent training.

Key Insights and Ideas

  • Decoupling Cloning from Translation: PVM fundamentally separates the voice cloning aspect from the core translation task. This allows for the use of pre-consented voices, sidestepping the ethical and legal issues of direct voice cloning.
  • Privacy by Design: The framework is built with privacy as a primary consideration, baking regulatory compliance directly into the S2ST pipeline.
  • Efficiency Gains: By avoiding direct cross-lingual voice cloning, PVM can lead to significant reductions in system run-time, especially in multi-speaker scenarios.
  • Improved Naturalness: The approach can paradoxically lead to more natural-sounding synthesized speech by leveraging monolingual TTS on matched preset-voices, rather than complex cross-lingual cloning.
  • Dataset Importance: The creation of CGDD highlights the need for diverse and well-annotated datasets to train robust emotion and gender classifiers, which are crucial for the PVM framework's success.

Key Results

  • Robustness: GEMO-Match classifiers trained on the CGDD dataset showed better generalization and accuracy compared to those trained solely on RAVDESS. For instance, gender classifiers achieved 97% accuracy on CGDD versus 94% on RAVDESS.
  • Multilingual Naturalness: PVM, implemented with GEMO-Match, significantly improved the output naturalness of S2ST systems. When translating from English to French and German, PVM achieved higher NISQA scores (e.g., 3.69 for German output vs. 3.41 for direct XTTS cloning).
  • Run-time Efficiency: PVM (specifically GEMO-Match with StyleTTS2) demonstrated significantly faster processing times compared to state-of-the-art deep multilingual voice cloning models like VALL-E X and XTTS. Auxiliary modules for PVM and OpenVoice achieved average run-times of 0.61 and 0.52 seconds, respectively, and PVM showed better scaling in multi-speaker scenarios.

Limitations and Future Work

  • Classifier Dependency: GEMO-Match requires training three unique classifiers for each source language, which can be data-intensive and complex to manage.
  • Feature Granularity: The current gender-emotion features might be insufficient for capturing the full nuance of speaker variance, necessitating more sophisticated feature extraction for closer matches.
  • Loss of Background Ambience: PVM does not inherently preserve background noise or ambience from the original input, which can be important in some applications.
  • Error Propagation: As a cascaded system, PVM is susceptible to error propagation from its constituent modules.

Future work suggested includes:

  • Developing PVM instances that don't rely on labeled data, potentially using self-supervised learning for robust internal speech representations.
  • Exploring cascaded voice cloning TTS systems that use an encoded representation of voices in the Preset-Voice Library, further enhancing privacy and efficiency.
  • Investigating methods to incorporate background ambience information.

Summary updated

BibTeX
@inproceedings{platnick2024preset,
  author = {Daniel Platnick and Bishoy Abdelnour and Eamon Earl and Rahul Kumar and Zahra Rezaei and Thomas Tsangaris and Faraj Lagum},
  title = {{Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure