TL;DR
Standard Large Language Models (LLMs) are built for static "read-at-once" processing, making them ill-suited for real-time, continuous data streams like live video or speech. This survey paper provides the first unified framework and taxonomy for "Streaming LLMs," breaking them down into output, sequential, and concurrent paradigms to guide future research in real-time interactive AI.
What Problem Does This Paper Address?
Traditional LLMs operate on a batch-processing paradigm: they ingest a complete static prompt, compute the Key-Value (KV) cache during a "prefill" phase, and then autoregressively decode tokens. However, real-world applications—such as live translation, streaming video analysis, and interactive robotics—deal with information that arrives incrementally and may be unbounded in length.
Furthermore, the term "streaming LLM" has become heavily overloaded in recent literature. It is often conflated across distinct concepts, such as token-by-token generation, chunk-wise encoding, or full-duplex interaction. The authors address this fragmentation by establishing a formal mathematical definition and a structured roadmap to transition LLMs from static inference to dynamic interaction.
Methods
As a comprehensive survey, the paper introduces a formal framework to unify the field, categorizing streaming LLMs into three distinct levels based on data flow and interaction concurrency:
- Output-Streaming LLMs: These models process static inputs but emit outputs progressively (token-wise, block-wise, or via refinement-based denoising).
- Sequential-Streaming LLMs: These models process dynamic, streaming inputs incrementally (avoiding quadratic re-computation) but wait until the input stream finishes before generating a response. Key techniques here involve incremental encoding and context management (e.g., KV cache eviction and compression).
- Concurrent-Streaming LLMs: The ultimate goal—simultaneously reading streaming inputs and writing outputs in a full-duplex manner.
The authors detail how concurrent streaming requires solving unique structural conflicts (like attention contention and position-ID overlaps) through architecture adaptations (re-encoding, concatenation, interleaving, or grouping) and learning interaction policies (rule-based, supervised fine-tuning, or reinforcement learning) to dynamically schedule when the model should listen versus when it should speak.
Datasets
As a survey paper, the authors do not introduce new datasets or train models. Instead, they compile and categorize existing benchmarks used across different modalities (text, speech, video) and tasks (such as streaming QA, translation, and tool usage). They highlight specialized evaluation benchmarks like SVbench for temporal multi-turn dialogues in streaming video understanding.
Key Insights and Ideas
- The "Read-While-Talking" Conflict: Moving from static to concurrent streaming isn't just an engineering challenge; it introduces fundamental architectural breaks. When new inputs arrive while a model is generating text, it triggers attention contention (ambiguity in causal history) and position-ID conflicts (inputs and outputs fighting for the same sequence index).
- Grouping over Interleaving: Among the architectural adaptations for concurrent streaming, grouped streaming—which partitions inputs and outputs into separate groups with independent position IDs—shows great promise in preserving pre-trained model performance while maximizing parallelism and efficiency.
- From Rules to RL: Early streaming systems relied on rigid, pre-defined rules (like the "Wait-k" policy in translation). The field is rapidly shifting toward Reinforcement Learning (RL) policies that treat interaction as sequential decision-making, allowing models to proactively decide when to pause, listen, or intervene based on real-time uncertainty.
Key Results
The paper synthesizes a vast body of recent literature into a clean, unified taxonomy. It maps out how landmark methods (e.g., StreamingLLM for memory efficiency, Qwen3-Omni for concurrent multimodal processing, and Simul-LLM for translation) fit into the broader streaming ecosystem. It provides researchers with a structured roadmap showing which challenges are shared across paradigms and which are unique to full-duplex interaction.
Limitations and Future Work
The authors note that their survey focuses primarily on high-level design principles and algorithmic paradigms for decoder-only LLMs, leaving out low-level system optimizations and hardware-specific deployment constraints.
For future work, they propose several critical directions:
- Alternative Concurrent Paradigms: Exploring block-wise or semi-autoregressive generation within streaming frameworks to reduce latency.
- Omni-Modal Continuous Streams: Expanding beyond text and audio to handle parallel, synchronized video-audio streams in highly dynamic environments.
- Deep Concurrency Levels: Moving from 2-level concurrency (listening-while-speaking) to multi-level asynchronous processing, such as 4-level streaming that concurrently handles perceiving, reasoning, tool-using, and generation.
Summary updated