Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems

Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri, Luisa BentivogliarXiv 2025 · DOI · arXiv

Introduction

Streaming Speech-to-Text Translation (StreamST) represents a significant challenge in machine translation: the ability to translate spoken language in real-time as a speaker continues to talk. This task requires balancing the inherent trade-off between latency (the delay before a translation appears) and quality (the accuracy of the output). While the research community has historically relied on tools like SimulEval, these legacy systems face several limitations. They are often unmaintained, biased toward incremental decoding paradigms, focused on short-form audio snippets, and lack integrated tools for live demonstrations.

To address these gaps, we introduce Simulstream, a unified, open-source toolkit designed for the evaluation and demonstration of StreamST systems. Simulstream supports both incremental and re-translation paradigms, handles long-form audio, and provides a ready-to-use web interface for real-world testing.

System Architecture

Simulstream is built on a full-duplex WebSocket client-server architecture, designed for flexibility and scalability:

  • The Server: This component manages a pool of "speech processors." It receives audio chunks (encoded as 16-bit integers) and returns JSON objects containing transcripts or translations.
  • The Clients: The toolkit includes a command-line client for high-throughput batch evaluation and an HTTP/web client for interactive, live demonstrations.
  • Speech Processors: The framework includes several pre-built processors, such as Sliding Window (a re-translation approach), StreamAtt (an incremental approach using attention scores), and wrappers for Voice Activity Detection (VAD) or legacy SimulEval agents.

A key feature of the framework is its ability to track token deletions. This allows researchers to quantify "flickering"—the visual instability that occurs when a re-translation system revises its previous output as more context becomes available.

Evaluation Methodology

Simulstream was evaluated using the MuST-C dataset across eight language pairs (English to German, Spanish, French, Italian, Dutch, Portuguese, Romanian, and Russian). Unlike standard evaluations that focus on isolated sentences, this study processed entire talks (averaging 10 minutes each) to assess performance on unsegmented, long-form audio.

The toolkit introduces "computational-aware" latency metrics. While traditional metrics focus on algorithmic delay, Simulstream measures the actual time hardware takes to process the model, providing a realistic view of whether a system is truly "real-time."

Key Insights and Results

Comparing Translation Paradigms

Simulstream enables the first fair comparison between re-translation (common in industrial applications) and incremental decoding (common in academic research). The study found that while re-translation often yields higher quality by allowing the model to correct itself, it introduces flickering. The toolkit quantifies this using Normalized Erasure (NE).

The Role of Voice Activity Detection (VAD)

Integrating a VAD wrapper into re-translation systems proved highly effective. It reduced flickering by approximately 8x and lowered computational costs by 2-3x. Although this resulted in a slight decrease in translation quality, the stability gains for the end-user were substantial.

Model Performance

In head-to-head testing, the Canary v2 model significantly outperformed SeamlessM4T v1 in translation quality (measured by COMET and BLEU) while maintaining comparable latency. The StreamAtt incremental strategy emerged as a top performer for optimizing the quality-latency trade-off, offering high stability (zero flickering) and high quality, albeit at a higher computational cost than simpler methods.

Real-Time Readiness

Most tested configurations achieved a Real Time Factor (RTF) well below 1.0. This confirms that the systems can process audio faster than it is spoken, validating the toolkit's readiness for live deployment scenarios.

Conclusion and Future Work

Simulstream fills a critical void in the StreamST research ecosystem by providing a robust framework for long-form evaluation and live demos. While the current results are promising, future work will focus on expanding the library of out-of-the-box processors and conducting extensive human user studies to better understand the subjective impact of flickering. Additionally, exploring hardware scaling for high-concurrency production environments remains a priority for the project's evolution.

Summary updated

Abstract

Streaming Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming speech under strict latency constraints, demanding models that balance low latency with high translation quality. Despite rapid progress, evaluation remains fragmented across existing frameworks, which make different assumptions about how systems operate - for example, whether they process continuous speech or short pre-segmented audio, and whether they support output revision (retranslation) or not (incremental). For instance, SimulEval, the most widely used framework, supports only incremental decoding, assumes short segmented inputs, and lacks a native support for system demonstrations. As a result, comparing systems fairly and consistently across studies remains challenging, with no unified solution for benchmarking and interactive demonstration. To address this gap, we introduce simulstream, the first open-source framework for StreamST evaluation and demonstration. It supports both incremental and re-translation decoding on long-form speech, provides fine-grained logging for quality and latency evaluation, and includes an interactive web interface for real-time visualization and comparison.
BibTeX
@misc{gaido2025simulstream,
  author = {Marco Gaido and Sara Papi and Mauro Cettolo and Matteo Negri and Luisa Bentivogli},
  title = {{Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems}},
  year = {2025},
  eprint = {2512.17648},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.17648},
}
Figure