Introduction
Streaming Speech-to-Text Translation (StreamST) represents a significant challenge in machine translation: the ability to translate spoken language in real-time as a speaker continues to talk. This task requires balancing the inherent trade-off between latency (the delay before a translation appears) and quality (the accuracy of the output). While the research community has historically relied on tools like SimulEval, these legacy systems face several limitations. They are often unmaintained, biased toward incremental decoding paradigms, focused on short-form audio snippets, and lack integrated tools for live demonstrations.
To address these gaps, we introduce Simulstream, a unified, open-source toolkit designed for the evaluation and demonstration of StreamST systems. Simulstream supports both incremental and re-translation paradigms, handles long-form audio, and provides a ready-to-use web interface for real-world testing.
System Architecture
Simulstream is built on a full-duplex WebSocket client-server architecture, designed for flexibility and scalability:
- The Server: This component manages a pool of "speech processors." It receives audio chunks (encoded as 16-bit integers) and returns JSON objects containing transcripts or translations.
- The Clients: The toolkit includes a command-line client for high-throughput batch evaluation and an HTTP/web client for interactive, live demonstrations.
- Speech Processors: The framework includes several pre-built processors, such as Sliding Window (a re-translation approach), StreamAtt (an incremental approach using attention scores), and wrappers for Voice Activity Detection (VAD) or legacy SimulEval agents.
A key feature of the framework is its ability to track token deletions. This allows researchers to quantify "flickering"—the visual instability that occurs when a re-translation system revises its previous output as more context becomes available.
Evaluation Methodology
Simulstream was evaluated using the MuST-C dataset across eight language pairs (English to German, Spanish, French, Italian, Dutch, Portuguese, Romanian, and Russian). Unlike standard evaluations that focus on isolated sentences, this study processed entire talks (averaging 10 minutes each) to assess performance on unsegmented, long-form audio.
The toolkit introduces "computational-aware" latency metrics. While traditional metrics focus on algorithmic delay, Simulstream measures the actual time hardware takes to process the model, providing a realistic view of whether a system is truly "real-time."
Key Insights and Results
Comparing Translation Paradigms
Simulstream enables the first fair comparison between re-translation (common in industrial applications) and incremental decoding (common in academic research). The study found that while re-translation often yields higher quality by allowing the model to correct itself, it introduces flickering. The toolkit quantifies this using Normalized Erasure (NE).
The Role of Voice Activity Detection (VAD)
Integrating a VAD wrapper into re-translation systems proved highly effective. It reduced flickering by approximately 8x and lowered computational costs by 2-3x. Although this resulted in a slight decrease in translation quality, the stability gains for the end-user were substantial.
Model Performance
In head-to-head testing, the Canary v2 model significantly outperformed SeamlessM4T v1 in translation quality (measured by COMET and BLEU) while maintaining comparable latency. The StreamAtt incremental strategy emerged as a top performer for optimizing the quality-latency trade-off, offering high stability (zero flickering) and high quality, albeit at a higher computational cost than simpler methods.
Real-Time Readiness
Most tested configurations achieved a Real Time Factor (RTF) well below 1.0. This confirms that the systems can process audio faster than it is spoken, validating the toolkit's readiness for live deployment scenarios.
Conclusion and Future Work
Simulstream fills a critical void in the StreamST research ecosystem by providing a robust framework for long-form evaluation and live demos. While the current results are promising, future work will focus on expanding the library of out-of-the-box processors and conducting extensive human user studies to better understand the subjective impact of flickering. Additionally, exploring hardware scaling for high-concurrency production environments remains a priority for the project's evolution.
Summary updated