TL;DR
SlackServe is a novel serving system designed for real-time streaming video generation using Autoregressive Diffusion Transformers (AR-DiTs). By treating "playout slack"—the buffer of generated video before playback catches up—as a unified control signal, it dynamically reallocates GPU resources and adjusts video quality on the fly. This approach drastically reduces video playback stalls and initial wait times while maintaining high visual quality.
What Problem Does This Paper Address?
Traditional video generation models (like Sora or Stable Video Diffusion) operate in an offline, "one-shot" fashion: you provide a prompt, wait several minutes, and receive the full video file. This doesn't work for interactive, real-time applications like AI live streaming or real-time visual effects.
To solve this, researchers recently developed Autoregressive Diffusion Transformers (AR-DiTs), which generate video chunk-by-chunk, allowing users to stream the video as it's being made. However, existing AI serving infrastructure is built for Large Language Models (LLMs) or offline image/video generation, optimizing for fixed metrics like throughput or latency per token.
Streaming video introduces a completely different challenge: playout continuity. The system's primary goal is to ensure that video generation always stays ahead of the user's playback timeline. If generation falls behind, the playable buffer—termed playout slack—is exhausted, and the user experiences a jarring pause or stall.
Methods
The authors present SlackServe, the first serving system built specifically around dynamic playout slack and adjustable fidelity. It decouples the serving architecture into three asynchronous planes:
- Control Plane (Slack-Driven Resource Reallocation & Fidelity Selection):
- Three-Tier Priority Queues: Classifies video streams into Urgent, Normal, or Relaxed tiers based on their playout slack, prioritizing urgent streams locally.
- Cross-Worker Re-homing: Dynamically migrates urgent streams away from overloaded GPUs to underutilized ones.
- Elastic Sequence Parallelism (Elastic SP): As a last resort for highly critical streams, it temporarily "borrows" a neighboring GPU to split the computational load of a single chunk using sequence parallelism.
- Bi-Modal Pareto Routing (BMPR): Instead of keeping video quality static, BMPR dynamically adjusts four "fidelity knobs" (denoising steps, attention sparsity, KV window size, and quantization precision) per chunk. If playout slack is healthy, it maximizes quality; if slack is tight, it drops to a lower-fidelity configuration to speed up generation, while strictly enforcing a global quality floor.
- State Plane: Manages the paged Key-Value (KV) cache at a frame-level granularity and uses an asynchronous transfer engine to stream state data layer-by-layer across GPUs, ensuring that migrations don't block active computations.
- Execution Plane: Dispatches and executes the video chunks on assigned GPU workers according to the Control Plane's decisions.
[ Playout Slack Signal ]
│
▼
┌─────────────────┐
│ Control Plane │
└────────┬────────┘
│
┌──────────────┴──────────────┐
▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│ Resource Reallocation │ │ Fidelity Selection │
│ (Queues, Re-homing, SP) │ │ (BMPR) │
└───────────────────────────┘ └───────────────────────────┘
Datasets
The system was evaluated using all 946 prompts from VBench, a comprehensive benchmark suite for video generative models that evaluates visual quality, temporal consistency, and semantic alignment.
To simulate real-world conditions, the authors constructed five distinct evaluation workloads:
- Steady: Streams arriving via a standard stationary Poisson process.
- Burst: Simulates flash crowds by forcing 10% of all streams to arrive simultaneously at specific intervals.
- Prompt-switch: Simulates users dynamically changing prompts mid-stream, resetting playout slack.
- Pause: Simulates users pausing playback, allowing the generation buffer to accumulate.
- Proprietary trace: A scaled, non-stationary enterprise trace reflecting realistic, messy arrival patterns.
Key Insights and Ideas
- The "Aha" Moment on KV Cache Propagation: The authors discovered that lowering the quality/precision of past video chunks has a surprisingly negligible impact on the visual quality of the current chunk. However, lowering the quality of the current chunk drastically reduces its generation latency. This means the system can aggressively downshift a single urgent chunk to avoid a playback stall without permanently ruining the quality of the rest of the video stream.
- Dynamic, Bidirectional SLOs: Unlike text generation where the target latency (Time Per Output Token) is fixed, streaming video has a moving target. Playout slack is bidirectional: resource allocation changes a stream's slack, and that slack in turn dictates future resource allocation.
- Timeliness > Perfection: In live streaming, an acceptable, slightly lower-fidelity chunk delivered on time is infinitely better than a flawless, high-fidelity chunk delivered late (which causes a visible stall).
Key Results
SlackServe was tested on a cluster of 16 NVIDIA H100 GPUs against state-of-the-art baselines like StreamDiffusionV2 and TridentServe.
- Improved Quality of Experience (QoE): SlackServe improved the Continuous Play Ratio (the fraction of chunks delivered before their deadline) by 1.64× to 3.29× across workloads.
- Reduced Wait Times: It reduced the Time to First Chunk (TTFC) by 1.61× to 9.65×, meaning users start watching the video much faster.
- Preserved Visual Quality: The overall drop in VBench quality scores was under 0.6%, proving that the system successfully confines quality degradation to brief, unnoticeable moments during heavy system overload.
- Shorter Stalls: In case studies, SlackServe reduced the average stall duration from 470–783 ms down to just 236 ms, alongside an 80%+ reduction in stall frequency per stream.
Limitations and Future Work
While SlackServe marks a major milestone for streaming video infrastructure, the authors note a few limitations:
- Static Profile Frontiers: The system currently relies on offline profiling to map out the speed-quality tradeoffs (Pareto frontiers). If a model behaves unexpectedly at runtime, these heuristics might become sub-optimal.
- Homogeneous Streams: The current implementation assumes all video streams share similar resolutions and playout rates.
- Future Directions: The team plans to extend SlackServe to support multi-task and multi-resolution serving environments, where different users might request varying aspect ratios, frame rates, and generation objectives simultaneously.
Summary updated