Abstract
Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, with increasing integration into meeting and streaming platforms. Despite this, SimulS2S remains underexplored in research, where current solutions often rely on resource-intensive training procedures and operate on short-form, pre-segmented utterances, failing to generalize to continuous speech. To bridge this gap, we propose SimulU, the first training-free policy for long-form SimulS2S. SimulU adopts history management and speech output selection strategies that exploit cross-attention in pre-trained end-to-end models to regulate both input history and output generation. Evaluations on MuST-C across 8 languages show that SimulU achieves a better or comparable quality-latency trade-off against strong cascaded models. By eliminating the need for ad-hoc training, SimulU offers a promising path to end-to-end SimulS2S in realistic, long-form scenarios.
Introduction
Simultaneous speech-to-speech translation (SimulS2S) represents the "universal translator" of science fiction, yet it remains a significant technical challenge. Most existing research focuses on "short-form" translation, where systems process pre-segmented audio clips under 30 seconds. However, real-world applications like live streaming or international meetings involve continuous, unsegmented speech.
Current approaches typically suffer from two main drawbacks: high resource requirements for training specialized policies and the "text bottleneck" inherent in cascaded systems (ASR → MT → TTS). Cascaded systems often introduce excessive latency and lose the original speaker's prosody. SimulU addresses these issues by introducing a training-free policy that repurposes the internal cross-attention mechanisms of pre-trained models, such as SeamlessM4T, to handle long-form audio "out of the box."
Methodology: The SimulU Workflow
The core of SimulU is an "onlinization" process that transforms an offline model into a streaming one. This is achieved through a six-step workflow that manages both input history and output generation without additional training.
- Audio Acquisition: Incoming speech is partitioned into small segments and appended to a rolling speech history.
- Hypothesis Generation: The system generates intermediate text representations from the accumulated audio.
- Stable Hypothesis Selection: Using cross-attention scores, the system identifies "stable" tokens—those strongly aligned with processed audio frames and unlikely to change as more audio arrives. This serves as the "read/write" decision.
- History Selection: To maintain efficiency during long-form sessions, SimulU uses a sliding window (Word History, $WH$) and attention scores to prune audio frames corresponding to discarded text.
- Speech Unit Generation: Stable text tokens are converted into speech units.
- Speech Hypothesis Selection: Finally, text-unit cross-attention is used to select and emit the synthesized speech units that correspond to the current stable hypothesis.
Experimental Evaluation
SimulU was evaluated using the MuST-C v1.0 corpus, focusing on English-to-8-language directions. To simulate realistic streaming, the researchers provided entire continuous TED Talks as input rather than pre-cut snippets.
Key Results
- Translation Quality: SimulU achieved superior ASR-BLEU scores in 6 out of 8 languages compared to strong baselines. Its end-to-end nature makes it more robust to the context starvation that often degrades cascaded models.
- Latency and Stability: The system maintained a "Start Offset" of 1 to 2 seconds, comparable to human interpreters. Furthermore, it demonstrated lower latency variance than cascaded models, providing a more consistent "flow" of translated speech.
- Controllability: By adjusting the cut-off frame hyperparameter ($f$), the system can be tuned to prioritize either lower latency or higher translation quality.
Conclusion
SimulU demonstrates that pre-trained transformer models already possess the internal knowledge necessary for effective simultaneous translation policies. By leveraging cross-attention scores, SimulU provides a high-quality, low-latency solution for long-form SimulS2S without the need for expensive retraining. Future work will focus on improving computational efficiency for incremental decoding and testing the system's robustness in noisy, multi-speaker environments.
Summary updated