StreamStudio: Exploring Video Authoring with Streaming Autoregressive Generation

Boyu Li, Zhe Lin, Eli Shechtman, Ding Li, Hongbo Fu, Yuqian ZhouarXiv 2026 · arXiv

TL;DR

StreamStudio is a novel interactive video authoring system that shifts generative video from a "prompt-and-wait" batch process to a real-time, steerable stream. By leveraging autoregressive models and a Git-like state management system, it allows creators to watch videos emerge frame-by-frame and intervene at any moment to retake, branch, or edit the content while maintaining perfect temporal continuity.

What Problem Does This Paper Address?

Most current generative video tools are built around diffusion models that use bidirectional temporal attention. This creates a "black box" workflow: a user submits a prompt, waits for the entire clip to render, and only then sees the result.

This approach has two major drawbacks for creators. First, it lacks immediacy; you can't stop a generation halfway through if it starts going off the rails. Second, it lacks granularity; if the first three seconds of a clip are perfect but the last two are bad, you usually have to regenerate the whole thing, often losing the parts you liked. StreamStudio addresses this by treating video generation as an ongoing, redirectable stream rather than a finished artifact.

Methods

The authors introduce a system architecture designed for streaming autoregressive generation. Unlike standard diffusion, which synthesizes a whole clip at once, autoregressive models generate video in sequential chunks.

Key technical components include:

  • Inference State Management: This is the system's "secret sauce." To allow a user to "retake" a shot from a specific frame, the system doesn't just save the pixels; it saves a checkpoint of the model's internal latent states, including KV (Key-Value) caches. This ensures that when a new branch starts, the motion and context remain continuous with the previous frames.
  • Git-like Branching: The system organizes different generation attempts as connected branches on a canvas. This allows users to explore multiple "what-if" scenarios without overwriting their previous work.
  • Dual-Level Interface:
    • Overview Canvas: For high-level project organization, managing assets like reference images and different video shots.
  • Shot Editor: A focused workspace featuring a live preview, a logical "Shot Canvas" for branching, and a multi-track timeline for temporal alignment.
The StreamStudio Shot Editor interface showing the canvas (a), preview (b), controls (c), and timeline (d).

Datasets

As a system-focused paper, StreamStudio does not introduce new training datasets. Instead, it employs a "plug-and-play" architecture using several state-of-the-art pre-trained models:

  • Image Generation: FLUX.2 Klein 4B.
  • Video Generation: LongLive 2.0 5B (specifically chosen for its real-time autoregressive capabilities).
  • Video Editing: JoyAI-Video-Edit and EditStream.

Key Insights and Ideas

  • Video as a Recording, Not a Render: The authors propose a paradigm shift where video creation feels more like recording an audio track. You can "punch in," "rewind," and "overdub" specific sections as the stream develops.
  • The Necessity of Latent Context: A crucial insight is that visual continuity cannot be maintained by pixels alone. To branch a video seamlessly, the system must restore the underlying causal inference state.
  • Co-evolution of Intent: By providing immediate feedback, the system supports the natural creative process where a creator's intent evolves as they see the work taking shape.
Technical breakdown of how the system restores inference states to allow for seamless branching and retakes.

Key Results

The paper demonstrates the system's effectiveness through a complex multi-step workflow involving a Siberian husky.

  • Incremental Construction: The user successfully builds a shot piece-by-piece (e.g., "husky blinking" followed by "husky turning around").
  • Non-Destructive Exploration: The user explores two different continuations from the same midpoint—one where the dog turns back to the camera and another where it exits the frame—keeping both versions available for comparison on the timeline.
  • Seamless Editing: The system successfully applies a "dog wearing a hat" edit to a specific branch, propagating the change forward while preserving the original footage.
The step-by-step workflow: from creating assets to generating, extending, retaking, and editing the video stream.

Limitations and Future Work

  • Model Performance: The interactivity of the system is strictly bounded by the inference speed of the underlying models. While "real-time" is the goal, current large models still have latency.
  • State Snapshots: Currently, retakes map to the beginning of a "chunk" boundary rather than every single frame, which slightly limits the precision of interventions.
  • Future Directions: The authors suggest moving toward richer multimodal interventions, such as allowing users to sketch or manipulate objects directly in the preview window during the generation stream.

Summary updated

Abstract

We present StreamStudio, a prototype system that explores a new authoring paradigm for streaming video generation. Instead of entering a prompt and waiting for a complete video before making changes, creators can interact with the generation as it unfolds: steering what comes next, returning to earlier moments, and exploring different continuations. In this way, StreamStudio shifts creative control from post-generation revision to continuous interaction with the generative process. Our contribution centers on this authoring paradigm and the interaction mechanisms that make streaming generation directly manipulable. StreamStudio is designed as a plug-and-play interface that can work with compatible generation and editing models, allowing its interaction model to evolve alongside advances in underlying video models. As streaming models become faster and easier to control in the future, such an interface may support more flexible and controllable video creation, including richer edits and multimodal interventions during generation.
BibTeX
@misc{li2026streamstudio,
  author = {Boyu Li and Zhe Lin and Eli Shechtman and Ding Li and Hongbo Fu and Yuqian Zhou},
  title = {{StreamStudio: Exploring Video Authoring with Streaming Autoregressive Generation}},
  year = {2026},
  eprint = {2610.38467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38467},
}
Figure