XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

Zunhai Su, Weihao Ye, Hansen Feng, Keyu Fan, Jing Zhang, Dahai Yu, Zhengwu Liu, Ngai Wong

TL;DR

XStreamVGGT introduces a tuning-free Key-Value (KV) cache compression framework designed specifically for streaming 3D vision transformers. By combining query-guided token pruning with dimension-adaptive quantization, it eliminates the problem of unbounded memory growth over long video sequences, slashing memory footprints by 4.42× and accelerating inference by 5.48× with virtually no loss in reconstruction accuracy.

What Problem Does This Paper Address?

State-of-the-art learning-based 3D vision models like StreamVGGT use transformers to perform real-time, streaming 3D reconstruction, depth estimation, and camera pose tracking from video feeds. To maintain temporal and geometric consistency across frames, these models rely on a Key-Value (KV) cache that stores representations of all past video tokens.

However, unlike text in Large Language Models (LLMs), video feeds generate a massive influx of visual tokens frame by frame. As a result, the KV cache grows linearly and boundlessly over time. This unbounded growth causes severe memory bloat and latency spikes, quickly leading to Out-of-Memory (OOM) crashes during long-horizon applications like robotics or autonomous driving.

Throughput degradation and OOM errors in standard models vs. XStreamVGGT

Methods

The authors present XStreamVGGT, a tuning-free pipeline that systematically compresses the KV cache through two tightly coupled steps:

  1. Query-Guided KV Cache Pruning: When the cache exceeds a predefined budget ($L_{max}$), the system drops redundant historical tokens. It aggregates the current frame's queries using average pooling and computes their similarity with cached keys to score token importance. Crucially, it always preserves the very first frame (to serve as a stable global geometric anchor) and the most recent frames. This mechanism avoids materializing full attention matrices, making it entirely compatible with hardware-optimized kernels like FlashAttention.
  2. Dimension-Adaptive KV Quantization: After pruning, the remaining tokens are compressed into a low-bit (INT4) format. Rather than applying uniform quantization, XStreamVGGT uses a mixed strategy tailored to the distinct mathematical distributions of vision tensors: per-channel quantization for Keys and per-token quantization for Values.
Figure 2: Overview of XStreamVGGT. Upon receiving a new input frame (Step 1), Queries from the global attention layer are aggregated via average pooling to form a compact representation, which is then matched against the Key to estimate token importance (Step 2). Guided by these Key-derived importance scores, low-importance historical KV pairs are selectively pruned, while KVs from the first frame are explicitly retained to preserve geometric consistency. The remaining high-importance KVs are concatenated with the first-frame KVs and the newly generated KVs from the current frame (Step 3). Following pruning, the KV cache is further compressed using dimension-adaptive quantization, employing per-channel Key quantization and per-token Value quantization to reduce the impact of outlier channels on quantization accuracy (Step 4). This results in a compact KV cache for efficient subsequent updates (Step 5).

Datasets

The authors evaluated XStreamVGGT across three core 3D vision tasks using standard, widely accepted benchmarks:

  • 3D Reconstruction: Evaluated on the 7-Scenes and NRGBD datasets.
  • Camera Pose Estimation: Evaluated on TUM Dynamics and ScanNet.
  • Monocular & Video Depth Estimation: Evaluated on Sintel, Bonn, and KITTI.

No new datasets were introduced; the experiments strictly followed established evaluation protocols from the 3D vision literature (such as Point3R and Monst3R).

Key Insights and Ideas

  • Asymmetric Outliers in 3D Vision Tensors: In LLMs, KV cache outliers typically appear across specific tokens. However, the authors discovered that in 3D vision transformers, Key tensors exhibit severe channel-wise outliers, whereas Value tensors remain relatively uniform. Standard per-token quantization causes these massive Key outliers to distort the quantization scale, destroying numerical precision. Shifting to per-channel quantization for Keys elegantly preserves accuracy.
  • The Power of the First-Frame Anchor: Retaining the first frame's KV cache explicitly acts as a vital geometric reference point. This design choice prevents spatial drift and cascading tracking errors, which typically plague aggressive pruning strategies in long-horizon video processing.
  • High Visual Redundancy: Visual streams contain massive spatial and temporal redundancies (e.g., static backgrounds or repeating textures). The paper demonstrates that only a small fraction of historical tokens are actually "heavy hitters" required to maintain high-fidelity 3D geometry.

Key Results

  • Massive Efficiency Gains: XStreamVGGT achieved a 4.42× reduction in GPU memory usage and a 5.48× inference speedup compared to the uncompressed StreamVGGT baseline.
  • OOM Elimination: While standard StreamVGGT suffered OOM crashes on an 80GB A100 GPU after 300–500 frames, XStreamVGGT maintained a completely flat, stable memory footprint and high throughput beyond 1,000 frames.
  • Near-Lossless Fidelity: On the NRGBD reconstruction dataset, Normal Consistency (NC) dropped by a negligible 1.4%. For monocular depth estimation on the Sintel and KITTI benchmarks, XStreamVGGT delivered identical performance to the uncompressed baseline, showing zero observable degradation.

Limitations and Future Work

The primary limitation is that the cache budget ($L_{max}$) is currently a fixed hyperparameter (set to 2,000 tokens in the experiments) that must be chosen before running inference.

As a next step, the authors suggest exploring adaptive cache budgets. Such a system would dynamically expand or contract the cache size on the fly, allocating more memory when encountering complex, highly dynamic scenes and aggressively compressing the cache during periods of slow motion or low texture variation.

Summary updated

Abstract

Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$\times$ and accelerating inference by 5.48$\times$, enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.
BibTeX
@misc{su2026xstreamvggt,
  author = {Zunhai Su and Weihao Ye and Hansen Feng and Keyu Fan and Jing Zhang and Dahai Yu and Zhengwu Liu and Ngai Wong},
  title = {{XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression}},
  year = {2026},
  eprint = {2601.01204},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.01204},
}
Figure