TL;DR
OmniTide is an algorithm-system co-design that enables efficient, real-time streaming of omni-modal LLMs (text, audio, and video) on consumer-grade devices. By using a "unit-based" abstraction to intelligently prune the KV cache and physically compacting memory to avoid fragmentation, it achieves up to $12.72\times$ kernel speedups and significant accuracy gains over standard sliding-window baselines.
What Problem Does This Paper Address?
Omni-modal models (like GPT-4o or Gemini Live) are moving toward continuous, full-duplex interaction where video, audio, and text streams are processed simultaneously. However, deploying these models on-device (e.g., a laptop or phone) hits a massive wall: KV cache growth.
As a session progresses, the Key-Value (KV) cache—the model's "memory" of past tokens—grows monotonically. Because multimodal data is incredibly dense (a few minutes of high-res video can generate 80K tokens), it quickly exhausts the limited RAM of consumer devices. For example, the authors found that just 6.25 minutes of streaming on an Apple M2 Pro consumed 23 GiB of memory, far exceeding the typical 16 GiB capacity.
Existing solutions fail in two ways:
- Algorithmic Gaps: Simple sliding windows (position-based) often "cut" through the middle of images or audio chunks, destroying context and tanking accuracy. More complex "heavy-hitter" selection methods (score-based) introduce too much computational overhead for real-time streaming.
- System Gaps: Even if you logically delete tokens, the remaining tokens are often scattered across memory. This "fragmentation" means hardware attention kernels still have to scan large physical spans, negating any speed benefits from pruning.
Methods
OmniTide introduces a two-part co-design centered around the "unit"—a temporal chunk of interleaved multimodal data.
1. OmniPick (The Algorithm)
OmniPick determines which tokens to keep based on three observations of "structural sparsity":
- Omni Sinks: Attention spikes at unit boundaries and modality transitions. OmniPick protects these "sinks" to maintain structural integrity.
- Unit Cohesion: It retains the most recent units in their entirety rather than pruning at the token level, ensuring images and audio from the same moment stay together.
- Modality Asymmetry: It uses separate retention budgets for different modalities (e.g., prioritizing audio over video) based on their information density.
2. OmniPage (The System)
OmniPage manages the physical layout of the KV cache to translate logical pruning into actual hardware speedups:
- Retention-Aware Paging: It groups tokens into physical pages based on their likelihood of being kept (e.g., "must-keep" vs. "evictable").
- Selective Migration: After pruning, it evaluates if moving the remaining tokens into a contiguous block will significantly reduce the "physical span" the GPU needs to scan. It only performs this move if the predicted speedup outweighs the cost of the memory copy.
Datasets
The authors evaluated OmniTide using three primary streaming benchmarks:
- StreamingBench: A benchmark for multiple-choice QA on long-form streaming video.
- SVBench: Focused on multi-turn video dialogue.
- LiveSports-3K-CC: A dataset for sports commentary requiring semantic alignment between video and text.
They tested these across three models: Qwen2.5-Omni (3B/7B) and MiniCPM-o-4.5 (9B), running on both NVIDIA RTX 4090 (CUDA) and Apple M2 Pro (Metal) architectures.
Key Insights and Ideas
- The "Unit" is the Right Abstraction: Treating multimodal streams as a sequence of atomic units (image + audio + text) prevents the semantic "hallucinations" that occur when token-level pruning accidentally deletes half of an image's representation.
- Sinks are Everywhere: Unlike text-only LLMs where attention sinks are mostly at the very beginning, omni-modal models show recurring sinks at every unit boundary.
- Physical Layout Matters: Logical sparsity is a "paper tiger" if the hardware kernels can't skip the empty holes. OmniPage's ability to compact the cache is what actually delivers the $12\times$ speedup.
Key Results
- Latency: OmniTide achieved up to $12.72\times$ kernel speedup and $2.40\times$ lower stream-loop latency compared to full-context baselines.
- Accuracy: On StreamingBench, OmniTide improved accuracy by up to 18.0 percentage points over standard sliding-window methods. It remained within 1 point of the "full context" accuracy while using a fraction of the memory.
- Memory Efficiency: It reduced the physical KV span by up to 26.7%. In one test, OmniTide processed a 241-second stream using only 0.164 GiB of KV cache, whereas the full-context baseline ballooned to 3.5 GiB in just 67 seconds.
Limitations and Future Work
- Watermark Tuning: The performance of OmniPick is sensitive to the "high/low watermark" settings (the thresholds for when to start and stop pruning). The authors found that the optimal settings vary by model, suggesting a need for automated tuning.
- Hardware Specifics: While tested on CUDA and Metal, the tile-aligned paging system needs to be tuned to the specific tile sizes of different GPU architectures to be most effective.
- Future Directions: The authors suggest integrating OmniTide with KV quantization or learned sparse attention mechanisms to further push the boundaries of infinite-context streaming.
Summary updated