OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming

Zongshang Shen, Wangsong Yin, Daliang Xu, Mengwei Xu, Xuanzhe LiuarXiv 2026 · arXiv

TL;DR

OmniTide is an algorithm-system co-design that enables efficient, real-time streaming of omni-modal LLMs (text, audio, and video) on consumer-grade devices. By using a "unit-based" abstraction to intelligently prune the KV cache and physically compacting memory to avoid fragmentation, it achieves up to $12.72\times$ kernel speedups and significant accuracy gains over standard sliding-window baselines.

What Problem Does This Paper Address?

Omni-modal models (like GPT-4o or Gemini Live) are moving toward continuous, full-duplex interaction where video, audio, and text streams are processed simultaneously. However, deploying these models on-device (e.g., a laptop or phone) hits a massive wall: KV cache growth.

As a session progresses, the Key-Value (KV) cache—the model's "memory" of past tokens—grows monotonically. Because multimodal data is incredibly dense (a few minutes of high-res video can generate 80K tokens), it quickly exhausts the limited RAM of consumer devices. For example, the authors found that just 6.25 minutes of streaming on an Apple M2 Pro consumed 23 GiB of memory, far exceeding the typical 16 GiB capacity.

Existing solutions fail in two ways:

  1. Algorithmic Gaps: Simple sliding windows (position-based) often "cut" through the middle of images or audio chunks, destroying context and tanking accuracy. More complex "heavy-hitter" selection methods (score-based) introduce too much computational overhead for real-time streaming.
  2. System Gaps: Even if you logically delete tokens, the remaining tokens are often scattered across memory. This "fragmentation" means hardware attention kernels still have to scan large physical spans, negating any speed benefits from pruning.
Figure 2. Memory and latency growth with full-history re- tention on MiniCPM-o-4.5. The model runs on Apple M2 Pro using Q4_K_M LLM weights and FP16 vision/audio pro- jectors, with Metal backend and FlashAttention enabled. Around 24 GB of unified memory is available for inference.

Methods

OmniTide introduces a two-part co-design centered around the "unit"—a temporal chunk of interleaved multimodal data.

1. OmniPick (The Algorithm)

OmniPick determines which tokens to keep based on three observations of "structural sparsity":

  • Omni Sinks: Attention spikes at unit boundaries and modality transitions. OmniPick protects these "sinks" to maintain structural integrity.
  • Unit Cohesion: It retains the most recent units in their entirety rather than pruning at the token level, ensuring images and audio from the same moment stay together.
  • Modality Asymmetry: It uses separate retention budgets for different modalities (e.g., prioritizing audio over video) based on their information density.

2. OmniPage (The System)

OmniPage manages the physical layout of the KV cache to translate logical pruning into actual hardware speedups:

  • Retention-Aware Paging: It groups tokens into physical pages based on their likelihood of being kept (e.g., "must-keep" vs. "evictable").
  • Selective Migration: After pruning, it evaluates if moving the remaining tokens into a contiguous block will significantly reduce the "physical span" the GPU needs to scan. It only performs this move if the predicted speedup outweighs the cost of the memory copy.
Comparison of OmniTide's unit-aware strategy against dense attention and other sparse methods.

Datasets

The authors evaluated OmniTide using three primary streaming benchmarks:

  • StreamingBench: A benchmark for multiple-choice QA on long-form streaming video.
  • SVBench: Focused on multi-turn video dialogue.
  • LiveSports-3K-CC: A dataset for sports commentary requiring semantic alignment between video and text.

They tested these across three models: Qwen2.5-Omni (3B/7B) and MiniCPM-o-4.5 (9B), running on both NVIDIA RTX 4090 (CUDA) and Apple M2 Pro (Metal) architectures.

Key Insights and Ideas

  • The "Unit" is the Right Abstraction: Treating multimodal streams as a sequence of atomic units (image + audio + text) prevents the semantic "hallucinations" that occur when token-level pruning accidentally deletes half of an image's representation.
  • Sinks are Everywhere: Unlike text-only LLMs where attention sinks are mostly at the very beginning, omni-modal models show recurring sinks at every unit boundary.
  • Physical Layout Matters: Logical sparsity is a "paper tiger" if the hardware kernels can't skip the empty holes. OmniPage's ability to compact the cache is what actually delivers the $12\times$ speedup.
Attention traces showing "omni sinks" at unit and modality boundaries across different models.

Key Results

  • Latency: OmniTide achieved up to $12.72\times$ kernel speedup and $2.40\times$ lower stream-loop latency compared to full-context baselines.
  • Accuracy: On StreamingBench, OmniTide improved accuracy by up to 18.0 percentage points over standard sliding-window methods. It remained within 1 point of the "full context" accuracy while using a fraction of the memory.
  • Memory Efficiency: It reduced the physical KV span by up to 26.7%. In one test, OmniTide processed a 241-second stream using only 0.164 GiB of KV cache, whereas the full-context baseline ballooned to 3.5 GiB in just 67 seconds.

Limitations and Future Work

  • Watermark Tuning: The performance of OmniPick is sensitive to the "high/low watermark" settings (the thresholds for when to start and stop pruning). The authors found that the optimal settings vary by model, suggesting a need for automated tuning.
  • Hardware Specifics: While tested on CUDA and Metal, the tile-aligned paging system needs to be tuned to the specific tile sizes of different GPU architectures to be most effective.
  • Future Directions: The authors suggest integrating OmniTide with KV quantization or learned sparse attention mechanisms to further push the boundaries of infinite-context streaming.

Summary updated

Abstract

On-device streaming omni-modal inference safeguards user privacy and eliminates prohibitive per-token API costs, but faces a critical bottleneck: the continuous influx of multimodal data rapidly exhausts constrained memory and compute budgets via monotonic KV cache growth. Existing sparse attention methods fall short, either incurring prohibitive online estimation latency or destroying interleaved cross-modal context, while failing to resolve physical memory fragmentation. We present OmniTide, the first algorithm-system co-design tailored for efficient on-device streaming omni-modal inference. Driven by the observation of modality-aware structural sparsity, OmniTide adopts a unit-based abstraction with two components: (1) At the algorithm level, OmniPick logically retains critical multimodal context based on unit boundaries and modality importance to preserve task accuracy; (2) At the system level, OmniPage physically partitions the cache by retention likelihood and dynamically compacts surviving sparse tokens, minimizing both memory fragmentation and data-movement overhead. Extensive evaluations across three streaming benchmarks and two consumer-device architectures show that OmniTide achieves up to $12.72\times$ kernel speedups and $2.40\times$ lower stream-loop latency. On StreamingBench, it improves accuracy by up to 18.0 percentage points over sliding-window baselines at comparable session cost. OmniPage further reduces the physical KV span by up to 26.7% relative to native logical eviction, unlocking real-time, infinite-context streaming on edge devices.
BibTeX
@misc{shen2026omnitide,
  author = {Zongshang Shen and Wangsong Yin and Daliang Xu and Mengwei Xu and Xuanzhe Liu},
  title = {{OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming}},
  year = {2026},
  eprint = {2609.34653},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.34653},
}
Figure