Unbabel Talk — Human Verified Translations for Voice Instant Messaging

Luís Bernardo, Mathieu Giquel, Sebastião Quintas, Paulo Dimas, Helena Moniz, Isabel TrancosoInterspeech 2019

TL;DR

Unbabel Talk is a novel application that bridges language barriers in voice instant messaging by combining state-of-the-art speech recognition, machine translation, and text-to-speech technologies with human verification. This allows users to send and receive voice messages in different languages, with a focus on ensuring high translation quality through Unbabel's community of translators.

What Problem Does This Paper Address?

The paper tackles the growing challenge of multilingual communication within voice instant messaging (IM). While voice messages offer convenience and richer emotional expression compared to text, they pose significant hurdles for users communicating across different languages. Recipients may struggle with understanding messages in unfamiliar languages, requiring them to rely on external translation tools or potentially miss crucial information. Unbabel Talk aims to solve this by providing a seamless, integrated solution for sending and receiving voice messages in multiple languages, ensuring accuracy and naturalness.

Methods

Unbabel Talk leverages a pipeline that integrates several key technologies:

  • Automatic Speech Recognition (ASR): Converts spoken words into text.
  • Machine Translation (MT): Translates the transcribed text from the source language to the target language.
  • Quality Estimation (QE): An internal Unbabel tool to assess the quality of the MT output.
  • Human Verification: For translations flagged as potentially low quality by QE, they are sent to Unbabel's community of human translators for correction.
  • Text-to-Speech (TTS): Synthesizes the translated text back into audio, using standard Amazon Polly voices in Version 1.

Version 2 is exploring advanced features like:

  • Transcription Correction by Human Translators: Allowing translators to directly correct the ASR output.
  • Voice Morphing: Using Tacotron + Wavenet TTS models to synthesize translated audio in the user's own voice, aiming for a more natural and personalized experience.

Datasets

Not explicitly discussed in the paper. The paper mentions that for voice morphing, a minimum of around 100 sentences in a specific voice are required to train TTS models, and a larger set would yield better quality. It also suggests that this speech material could ideally be obtained from the usage of the Unbabel Talk app with user permission.

Key Insights and Ideas

  • The Power of Human-in-the-Loop: The paper strongly emphasizes that while ASR, MT, and TTS are advancing rapidly, human verification remains crucial for ensuring high-quality, reliable translations, especially in sensitive communication scenarios like customer support.
  • Bridging the Gap in Voice IM: Unbabel Talk directly addresses a growing user need for seamless cross-lingual communication in a popular and increasingly adopted IM feature.
  • Future of Personalized Voice Communication: The exploration of voice morphing points towards a future where translated voice messages can retain the sender's unique vocal characteristics, significantly enhancing naturalness and user experience.
  • Challenges in Cross-Lingual Voice Synthesis: The paper acknowledges the significant technical hurdles in achieving realistic cross-lingual voice morphing, highlighting it as an active area of research.

Key Results

The paper focuses on the system's functionality and potential rather than presenting extensive quantitative results.

  • Version 1: Without human intervention, the pipeline (ASR, MT, TTS) takes approximately 4 seconds. Human verification, depending on the language pair and translator availability, can add 2 to 3 minutes to the process.
  • Version 2 (under development): The paper showcases a demonstration of Version 2 with voice morphing, trained on a limited set of ~100 sentences, to illustrate the achievable quality.

Limitations and Future Work

Limitations:

  • Latency of Human Verification: The most significant limitation identified is the time delay introduced by human verification, which can be substantial (2-3 minutes) and may not be suitable for real-time conversations.
  • Quality of Synthesized Voices: While Version 1 uses standard TTS voices, the paper acknowledges that they may not sound as natural as the user's own voice.
  • Challenges in Cross-Lingual Voice Morphing: Achieving natural-sounding voice morphing across different languages is a complex problem that requires further research. The current Tacotron + Wavenet system also has a notable synthesis time.

Future Work:

  • Reducing Latency: Continued efforts to optimize the translation pipeline and potentially explore asynchronous human feedback mechanisms to minimize delays.
  • Improving Voice Morphing: Further research into Tacotron + Wavenet and other TTS systems to achieve more natural and faster voice synthesis, especially in cross-lingual scenarios.
  • Expanding Language Support: Implicitly, the goal is to support a wide range of languages for both translation and voice synthesis.
  • Ethical Considerations: The paper mentions obtaining speech material with user permission and adhering to Unbabel's Code of Ethics, indicating an ongoing focus on responsible data handling.

Summary updated

BibTeX
@inproceedings{bernardo2019unbabel,
  author = {Luís Bernardo and Mathieu Giquel and Sebastião Quintas and Paulo Dimas and Helena Moniz and Isabel Trancoso},
  title = {{Unbabel Talk — Human Verified Translations for Voice Instant Messaging}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure