SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation

Amirbek Djanibekov, Luisa Bentivogli, Matteo Negri, Sara Papi

Abstract

Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, with increasing integration into meeting and streaming platforms. Despite this, SimulS2S remains underexplored in research, where current solutions often rely on resource-intensive training procedures and operate on short-form, pre-segmented utterances, failing to generalize to continuous speech. To bridge this gap, we propose SimulU, the first training-free policy for long-form SimulS2S. SimulU adopts history management and speech output selection strategies that exploit cross-attention in pre-trained end-to-end models to regulate both input history and output generation. Evaluations on MuST-C across 8 languages show that SimulU achieves a better or comparable quality-latency trade-off against strong cascaded models. By eliminating the need for ad-hoc training, SimulU offers a promising path to end-to-end SimulS2S in realistic, long-form scenarios.

Introduction

Simultaneous speech-to-speech translation (SimulS2S) represents the "universal translator" of science fiction, yet it remains a significant technical challenge. Most existing research focuses on "short-form" translation, where systems process pre-segmented audio clips under 30 seconds. However, real-world applications like live streaming or international meetings involve continuous, unsegmented speech.

Conceptual overview of simultaneous speech-to-speech translation with start offset latency.

Current approaches typically suffer from two main drawbacks: high resource requirements for training specialized policies and the "text bottleneck" inherent in cascaded systems (ASR → MT → TTS). Cascaded systems often introduce excessive latency and lose the original speaker's prosody. SimulU addresses these issues by introducing a training-free policy that repurposes the internal cross-attention mechanisms of pre-trained models, such as SeamlessM4T, to handle long-form audio "out of the box."

Methodology: The SimulU Workflow

The core of SimulU is an "onlinization" process that transforms an offline model into a streaming one. This is achieved through a six-step workflow that manages both input history and output generation without additional training.

The six-step SimulU workflow for on-the-fly simultaneous speech-to-speech translation.
  1. Audio Acquisition: Incoming speech is partitioned into small segments and appended to a rolling speech history.
  2. Hypothesis Generation: The system generates intermediate text representations from the accumulated audio.
  3. Stable Hypothesis Selection: Using cross-attention scores, the system identifies "stable" tokens—those strongly aligned with processed audio frames and unlikely to change as more audio arrives. This serves as the "read/write" decision.
  4. History Selection: To maintain efficiency during long-form sessions, SimulU uses a sliding window (Word History, $WH$) and attention scores to prune audio frames corresponding to discarded text.
  5. Speech Unit Generation: Stable text tokens are converted into speech units.
  6. Speech Hypothesis Selection: Finally, text-unit cross-attention is used to select and emit the synthesized speech units that correspond to the current stable hypothesis.

Experimental Evaluation

SimulU was evaluated using the MuST-C v1.0 corpus, focusing on English-to-8-language directions. To simulate realistic streaming, the researchers provided entire continuous TED Talks as input rather than pre-cut snippets.

Key Results

  • Translation Quality: SimulU achieved superior ASR-BLEU scores in 6 out of 8 languages compared to strong baselines. Its end-to-end nature makes it more robust to the context starvation that often degrades cascaded models.
  • Latency and Stability: The system maintained a "Start Offset" of 1 to 2 seconds, comparable to human interpreters. Furthermore, it demonstrated lower latency variance than cascaded models, providing a more consistent "flow" of translated speech.
  • Controllability: By adjusting the cut-off frame hyperparameter ($f$), the system can be tuned to prioritize either lower latency or higher translation quality.

Conclusion

SimulU demonstrates that pre-trained transformer models already possess the internal knowledge necessary for effective simultaneous translation policies. By leveraging cross-attention scores, SimulU provides a high-quality, low-latency solution for long-form SimulS2S without the need for expensive retraining. Future work will focus on improving computational efficiency for incremental decoding and testing the system's robustness in noisy, multi-speaker environments.

Summary updated

Abstract

Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, with increasing integration into meeting and streaming platforms. Despite this, SimulS2S remains underexplored in research, where current solutions often rely on resource-intensive training procedures and operate on short-form, pre-segmented utterances, failing to generalize to continuous speech. To bridge this gap, we propose SimulU, the first training-free policy for long-form SimulS2S. SimulU adopts history management and speech output selection strategies that exploit cross-attention in pre-trained end-to-end models to regulate both input history and output generation. Evaluations on MuST-C across 8 languages show that SimulU achieves a better or comparable quality-latency trade-off against strong cascaded models. By eliminating the need for ad-hoc training, SimulU offers a promising path to end-to-end SimulS2S in realistic, long-form scenarios.
BibTeX
@misc{djanibekov2026simulu,
  author = {Amirbek Djanibekov and Luisa Bentivogli and Matteo Negri and Sara Papi},
  title = {{SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation}},
  year = {2026},
  eprint = {2603.16924},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.16924},
}
Figure