In a standard conversation or translation task, an AI typically waits for a user to finish speaking before processing the input. This "turn-taking" approach is safe but introduces a significant delay that breaks the flow of natural interaction. To achieve real-time communication—similar to a professional human interpreter—systems must perform simultaneous translation, generating output while the source input is still being received.
This creates a fundamental tension: the system must decide whether to READ (wait for more context to ensure accuracy) or WRITE (generate output immediately to minimize latency).
The Read-Write Policy
The core of any simultaneous system is its "policy"—the logic that dictates when to commit to an output. As Fahim Dalvi et al. describe in Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation, early systems often used static policies, such as "wait-k," where the model waits for $k$ source tokens before it starts translating.
However, static policies struggle with language pairs that have divergent word orders, such as English (Subject-Verb-Object) and Japanese (Subject-Object-Verb). If the English verb appears at the end of a long clause, a static model might be forced to guess the action or wait so long that the latency becomes unacceptable. To solve this, Ruiqing Zhang et al. proposed adaptive segmentation policies in Learning Adaptive Segmentation Policy for Simultaneous Translation. These models learn to identify "Meaningful Units" (MUs), allowing the system to wait for just enough context to maintain grammatical correctness without unnecessary delay.
Predicting the Unspoken
When the system cannot afford to wait for the speaker to finish, it must "hallucinate" or predict the missing information. Ashkan Alinejad et al. introduced a PREDICT action in Prediction Improves Simultaneous Neural Machine Translation, which allows the agent to guess upcoming source words. By anticipating the end of a sentence, the model can begin the translation process earlier.
This prediction isn't just about words; it can also be syntactic. In Syntax-based Simultaneous Translation through Prediction of Unseen Syntactic Constituents, Yusuke Oda et al. demonstrated that predicting the structural components of a sentence (like a missing verb phrase in a parse tree) allows the model to produce more coherent word orders in real-time.
From Cascades to End-to-End Architectures
Traditionally, speech translation is "cascaded": an Automatic Speech Recognition (ASR) module transcribes audio, a Machine Translation (MT) module translates the text, and a Text-to-Speech (TTS) module generates audio. As noted by Nameer Hirschkind et al. in Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation, these cascades suffer from cumulative latency—each step must wait for the previous one to produce a result.
Modern research is shifting toward end-to-end models that map speech directly to the target language. Christian Huber et al. argue in End-to-End Evaluation for Low-Latency Simultaneous Speech Translation that these architectures eliminate the communication overhead between modules, offering a significant speed advantage. For example, the NAST-S2x framework proposed by Zhengrui Ma et al. uses non-autoregressive generation to process chunks of speech in parallel, achieving a 28x speedup over traditional step-by-step methods.
The Stability Challenge
A major side effect of low-latency translation is "flicker"—the tendency of a system to change its mind and revise previous words as more context arrives. While re-translation strategies can improve quality, they can be visually or auditorily jarring. Naveen Arivazhagan et al. addressed this in Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translation by using biasing and masking heuristics to stabilize the output, ensuring that once a word is shown to the user, it is rarely changed.
Furthermore, Jorge Iranzo-Sánchez points out in Going Beyond Your Expectations in Latency Metrics for Simultaneous Speech Translation that average latency is a poor metric for user experience. A system might have a low average delay but suffer from unpredictable "latency spikes" that confuse the listener. Tighter bounds on maximum delay are often more important for natural interaction than the highest possible accuracy score.
Full-Duplex Dialogue
The ultimate goal of this research direction is full-duplex interaction, where the AI can listen and speak simultaneously, handling interruptions just like a human. In A Full-duplex Speech Dialogue Scheme Based On Large Language Model, Peng Wang et al. show that by framing conversation management as a next-token prediction task within a Large Language Model (LLM), systems can cut response latency to under 300ms, enabling truly seamless, real-time human-agent dialogue.