The Challenge of Simultaneous Translation
Simultaneous speech-to-speech translation (S2ST)—often compared to the demanding task of a "UN interpreter"—is notoriously difficult for AI. The model must decide in real-time when to listen and when to start speaking, often before the speaker has finished their sentence.
Traditionally, training these models requires "word-level aligned data," which maps specific source audio segments to specific target words. Because this data is virtually non-existent in the real world, researchers typically rely on synthetic data generated via complex, hand-crafted heuristics. These heuristics are often brittle and highly language-specific, creating a major bottleneck for scaling S2ST systems to new languages.
Introducing Hibiki-Zero
Hibiki-Zero is a breakthrough in S2ST that eliminates the need for labor-intensive, word-level aligned training data. By relying entirely on "sentence-level" alignment—which is significantly easier to obtain—the model fundamentally simplifies the training pipeline. This heuristic-free approach allows Hibiki-Zero to scale seamlessly to diverse languages with varying grammatical structures, such as verb-final languages, without requiring custom alignment rules.
How It Works: Multistream Architecture and Reinforcement Learning
Hibiki-Zero is a decoder-only, multistream model based on the RQ-Transformer architecture. It processes several streams of information simultaneously: the input audio, an "Inner Monologue" (text translation), and the output audio tokens. Predicting text tokens alongside audio tokens acts as scaffolding, significantly stabilizing the audio generation and ensuring the spoken output matches the translated meaning.
The training pipeline consists of three main stages:
- Audio Pretraining: Learning to represent speech using the Mimi codec.
- Coarse Translation Training: Training on sentence-level aligned data using a "silence insertion" technique to create a rough, non-simultaneous starting point.
- Reinforcement Learning (The Core Innovation): The authors adapt Group Relative Policy Optimization (GRPO) to the multistream setting. Instead of relying on hand-coded "Wait-k" policies or rewarding the model only for the final translation, Hibiki-Zero uses Process Rewards. By calculating BLEU scores at multiple intermediate points during translation, the model faces natural pressure to reduce latency. If it waits too long, it misses out on early rewards; if it speaks too soon and makes a mistake, its BLEU score drops.
Key Results and Benchmarks
The model was trained on a massive 40,000-hour dataset covering French, Spanish, Portuguese, and German to English. To evaluate long-form speech translation, the researchers also released Audio-NTREX-4L, a new 45-hour multilingual benchmark synthesized with high-quality industry TTS.
Hibiki-Zero achieves state-of-the-art performance, outperforming existing systems like Meta’s Seamless and the original Hibiki model across several metrics:
- Translation Quality: On long-form tasks, Hibiki-Zero beats Seamless by over 5 points in ASR-BLEU.
- Voice Preservation: It demonstrates a massive lead in speaker identity preservation, scoring 61.3 in speaker similarity for French-to-English, compared to just 44.4 for Seamless.
- Latency: It achieves lower Length-Adaptive Average Lagging (LAAL), meaning it starts translating sooner without sacrificing accuracy.
- Adaptability: The model can be adapted to support a new input language (demonstrated with Italian) using less than 1,000 hours of speech data.
Human evaluators consistently gave Hibiki-Zero higher scores for speech naturalness and audio quality across all tested languages.
Limitations and Future Directions
While Hibiki-Zero is excellent at preserving the identity of the speaker's voice, it does not yet allow for fine-grained control over the "intensity" of the source language's accent in the translated output. Furthermore, the balance between quality and latency is currently static, having been "baked in" during the RL phase. Users cannot adjust a "latency slider" at inference time to make the model faster or more accurate on the fly.
Future work will explore providing accent-annotated samples during training to solve the accent control issue, as well as developing methods to make the quality-latency trade-off adjustable during real-time use.
Summary updated