Holistic Parallel Supervision for Expressive Speech to Speech Translation

Hao Zhang, Ismail Rasim Ulgen, Bismarck Bamfo Odoom, Rui Liu, Zihan Zhang, Berrak Sisman, Philipp KoehnarXiv 2026 · arXiv

TL;DR

Researchers have unlocked a new level of expressiveness in speech-to-speech translation (S2ST) by training models on professional dubbing data enhanced with voice conversion. By creating "holistic" training pairs that match in content, prosody, timing, and speaker identity, their new DirectS2ST model can translate not just the words, but the emotional weight and delivery of the original speaker.

What Problem Does This Paper Address?

Most speech-to-speech translation systems are "emotionally flat." They might get the words right, but they lose the speaker’s unique voice, the dramatic pauses, and the melodic intonation (prosody) that convey intent.

The bottleneck is data. Existing datasets usually fall into three categories, each with a flaw:

  1. Synthetic data: Reproduces the speaker's voice but lacks natural expressive delivery.
  2. Mined data: Scalable, but the source and target speakers are different people saying different things in different styles.
  3. Professional dubbing: Offers great human-performed expressiveness and timing, but the source and target are different actors.

Without "holistic" supervision—where the source and target speech are perfectly aligned across content, prosody, timing, and identity—models struggle to learn how to transfer expression across languages.

Methods

The authors propose a two-part solution: a novel data construction strategy called VC-DUB and a new end-to-end model called DirectS2ST.

1. VC-DUB: The Data Recipe

To create the perfect training pair, the team started with professional dubbing from TV shows (drama, comedy, action).

  • The Challenge: The original German actor and the English dubbing actor have different voices.
  • The Fix: They used Multilingual Voice Conversion (VC) to transform the English dubbing actor's voice to sound like the original German actor.
  • The Result: A training pair where the "source" is the voice-converted dub and the "target" is the original human recording. This ensures the model learns from real human performance while maintaining a consistent speaker identity.
Fig. 1. Overview of the holistic parallel supervision: (a) VC-DUB Construction; (b) Parallel Supervision; (c) DirectS2ST.

2. DirectS2ST: The Model

DirectS2ST is an end-to-end model inspired by the "Moshi" architecture. It doesn't just translate text; it predicts the entire acoustic representation (using Mimi codecs) of the target speech directly from the source.

  • Temporal Decoder: An autoregressive component that predicts the source text, target text, and the first layer of audio codes.
  • Depth Decoder: A non-autoregressive Transformer that fills in the remaining 15 layers of audio detail, conditioned on a speaker-identity embedding.

Datasets

  • VC-DUB (New): Constructed from professional dubbing for English-German (En-De) and English-Spanish (En-Es). It contains roughly 131k pairs for En-De and 79k for En-Es.
  • CVSS-T: A standard baseline dataset used for comparison, which uses synthetic voice-transferred targets.
  • mExpresso: An independent, high-quality expressive multilingual dataset used strictly for evaluation to ensure the models generalize well to new speakers and styles.

Key Insights and Ideas

  • Dubbing is a Goldmine: Professional dubbing is essentially a human-in-the-loop expressive translation. The actors have already done the hard work of matching the timing and emotion of the original performance; researchers just needed a way to fix the "speaker mismatch" via voice conversion.
  • Source-Side VC is Safer: The authors applied voice conversion to the source speech rather than the target. This prevents the model from learning to mimic the artifacts or "robotic" sounds that voice conversion models sometimes produce in the target output.
  • Direct Modeling Wins: While holistic supervision improved all models tested (including modular ones like TransVIP), the end-to-end DirectS2ST saw the biggest gains. This suggests that predicting the full audio representation at once allows the model to better exploit the rich alignment in the data.

Key Results

The researchers evaluated their approach using both automatic metrics and human listeners.

  • Prosody and Timing: DirectS2ST trained on VC-DUB achieved the highest scores in prosody similarity (A.PCP) and pause similarity across both En-De and En-Es directions.
  • Speaker Similarity: Applying VC to the dubbing data (VC-DUB vs. RAW-DUB) led to a massive jump in speaker similarity (Vsim) scores, proving that the identity gap was successfully bridged.
  • Human Evaluation: In a 5-point expressiveness MOS (Mean Opinion Score) test, DirectS2ST saw a staggering +1.29 gain when moving from standard synthetic data to the new holistic supervision.

Limitations and Future Work

  • Data Licensing: Because the training data comes from professional entertainment media, the authors cannot share the raw audio files, though they have released the scripts and processing thresholds to allow others to recreate the pipeline.
  • Lexical Accuracy: While DirectS2ST excels at how things are said, its raw translation accuracy (BLEU) is currently lower than models that use massive pre-trained text-translation backbones (like SeamlessM4T).
  • Future Directions: The authors suggest that scaling this approach and combining it with larger pre-trained encoders could bridge the gap between expressive delivery and perfect lexical translation.

Summary updated

Abstract

Expressiveness is central to speech-to-speech translation (S2ST), yet existing datasets rarely align source and target speech jointly in content, prosody, timing, and speaker identity. Consequently, current systems often rely on indirect conditioning or transfer mechanisms. To obtain this supervision, we introduce a training-data construction strategy that combines professional dubbing, which provides human-performed correspondence in content, prosody, and timing, with multilingual voice conversion to reduce speaker mismatch. We evaluate this supervision using existing expressive S2ST systems and DirectS2ST, an end-to-end method developed in this work that predicts the complete target-speech representation directly from source speech. Experiments on English-German and English-Spanish show that holistic parallel supervision improves expressive realization across all evaluated systems. DirectS2ST exhibits the largest gains in content, prosody, timing, speaker similarity, and human-rated expressiveness. To our knowledge, this work provides the first systematic investigation of holistic parallel supervision for expressive S2ST.
BibTeX
@misc{zhang2026holistic,
  author = {Hao Zhang and Ismail Rasim Ulgen and Bismarck Bamfo Odoom and Rui Liu and Zihan Zhang and Berrak Sisman and Philipp Koehn},
  title = {{Holistic Parallel Supervision for Expressive Speech to Speech Translation}},
  year = {2026},
  eprint = {2611.06010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06010},
}
Figure