From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models

Junlong Tong, Zilong Wang, YuJie Ren, Peiran Yin, Hao Wu, Wei Zhang, Xiaoyu Shen

TL;DR

Standard Large Language Models (LLMs) are built for static "read-at-once" processing, making them ill-suited for real-time, continuous data streams like live video or speech. This survey paper provides the first unified framework and taxonomy for "Streaming LLMs," breaking them down into output, sequential, and concurrent paradigms to guide future research in real-time interactive AI.

What Problem Does This Paper Address?

Traditional LLMs operate on a batch-processing paradigm: they ingest a complete static prompt, compute the Key-Value (KV) cache during a "prefill" phase, and then autoregressively decode tokens. However, real-world applications—such as live translation, streaming video analysis, and interactive robotics—deal with information that arrives incrementally and may be unbounded in length.

Furthermore, the term "streaming LLM" has become heavily overloaded in recent literature. It is often conflated across distinct concepts, such as token-by-token generation, chunk-wise encoding, or full-duplex interaction. The authors address this fragmentation by establishing a formal mathematical definition and a structured roadmap to transition LLMs from static inference to dynamic interaction.

Methods

As a comprehensive survey, the paper introduces a formal framework to unify the field, categorizing streaming LLMs into three distinct levels based on data flow and interaction concurrency:

  1. Output-Streaming LLMs: These models process static inputs but emit outputs progressively (token-wise, block-wise, or via refinement-based denoising).
  2. Sequential-Streaming LLMs: These models process dynamic, streaming inputs incrementally (avoiding quadratic re-computation) but wait until the input stream finishes before generating a response. Key techniques here involve incremental encoding and context management (e.g., KV cache eviction and compression).
  3. Concurrent-Streaming LLMs: The ultimate goal—simultaneously reading streaming inputs and writing outputs in a full-duplex manner.
Illustration of the three streaming LLM paradigms: Output, Sequential, and Concurrent streaming.

The authors detail how concurrent streaming requires solving unique structural conflicts (like attention contention and position-ID overlaps) through architecture adaptations (re-encoding, concatenation, interleaving, or grouping) and learning interaction policies (rule-based, supervised fine-tuning, or reinforcement learning) to dynamically schedule when the model should listen versus when it should speak.

Structural conflicts (attention contention and position-ID conflict) when adapting batch LLMs to concurrent streaming.

Datasets

As a survey paper, the authors do not introduce new datasets or train models. Instead, they compile and categorize existing benchmarks used across different modalities (text, speech, video) and tasks (such as streaming QA, translation, and tool usage). They highlight specialized evaluation benchmarks like SVbench for temporal multi-turn dialogues in streaming video understanding.

Key Insights and Ideas

  • The "Read-While-Talking" Conflict: Moving from static to concurrent streaming isn't just an engineering challenge; it introduces fundamental architectural breaks. When new inputs arrive while a model is generating text, it triggers attention contention (ambiguity in causal history) and position-ID conflicts (inputs and outputs fighting for the same sequence index).
  • Grouping over Interleaving: Among the architectural adaptations for concurrent streaming, grouped streaming—which partitions inputs and outputs into separate groups with independent position IDs—shows great promise in preserving pre-trained model performance while maximizing parallelism and efficiency.
  • From Rules to RL: Early streaming systems relied on rigid, pre-defined rules (like the "Wait-k" policy in translation). The field is rapidly shifting toward Reinforcement Learning (RL) policies that treat interaction as sequential decision-making, allowing models to proactively decide when to pause, listen, or intervene based on real-time uncertainty.

Key Results

The paper synthesizes a vast body of recent literature into a clean, unified taxonomy. It maps out how landmark methods (e.g., StreamingLLM for memory efficiency, Qwen3-Omni for concurrent multimodal processing, and Simul-LLM for translation) fit into the broader streaming ecosystem. It provides researchers with a structured roadmap showing which challenges are shared across paradigms and which are unique to full-duplex interaction.

Limitations and Future Work

The authors note that their survey focuses primarily on high-level design principles and algorithmic paradigms for decoder-only LLMs, leaving out low-level system optimizations and hardware-specific deployment constraints.

For future work, they propose several critical directions:

  1. Alternative Concurrent Paradigms: Exploring block-wise or semi-autoregressive generation within streaming frameworks to reduce latency.
  2. Omni-Modal Continuous Streams: Expanding beyond text and audio to handle parallel, synchronized video-audio streams in highly dynamic environments.
  3. Deep Concurrency Levels: Moving from 2-level concurrency (listening-while-speaking) to multi-level asynchronous processing, such as 4-level streaming that concurrently handles perceiving, reasoning, tool-using, and generation.

Summary updated

Abstract

Standard Large Language Models (LLMs) are predominantly designed for static inference with pre-defined inputs, which limits their applicability in dynamic, real-time scenarios. To address this gap, the streaming LLM paradigm has emerged. However, existing definitions of streaming LLMs remain fragmented, conflating streaming generation, streaming inputs, and interactive streaming architectures, while a systematic taxonomy is still lacking. This paper provides a comprehensive overview and analysis of streaming LLMs. First, we establish a unified definition of streaming LLMs based on data flow and dynamic interaction to clarify existing ambiguities. Building on this definition, we propose a systematic taxonomy of current streaming LLMs and conduct an in-depth discussion on their underlying methodologies. Furthermore, we explore the applications of streaming LLMs in real-world scenarios and outline promising research directions to support ongoing advances in streaming intelligence. We maintain a continuously updated repository of relevant papers at https://github.com/EIT-NLP/Awesome-Streaming-LLMs.
BibTeX
@inproceedings{tong2026from,
  author = {Junlong Tong and Zilong Wang and YuJie Ren and Peiran Yin and Hao Wu and Wei Zhang and Xiaoyu Shen},
  title = {{From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.498},
}
Figure