Fold the Loop: Decoding-Time KV Memory for Looped Vision-Language Models

Kun Zhou, Wenmian Yang, Zhensheng Wang, Yanzeng Li, Weijia JiaarXiv 2026 · arXiv

TL;DR

Looped Vision-Language Models (VLMs) achieve deeper reasoning by recycling the same neural modules multiple times, but this creates a massive memory bottleneck: the Key-Value (KV) cache grows with every "loop." FoldCache solves this by identifying that visual representations are redundant across these loops during decoding; by "folding" these states so that multiple recurrent steps share a single set of visual KV entries, FoldCache halves memory usage with virtually zero loss in accuracy.

What Problem Does This Paper Address?

Modern VLMs are increasingly using "looped" or recurrent architectures. Instead of just passing an image through a long sequence of unique layers, these models pass representations through the same weight-tied modules multiple times. This "recurrent depth" allows the model to "think longer" about an image without increasing the number of parameters.

However, there’s a catch: the KV Cache. During the decoding phase (when the model generates text token by token), it stores previously computed Key and Value states to avoid redundant calculations. In a looped VLM, every single pass through a module generates a new set of KV states for the same image tokens. If a model loops 8 times, the cache holds 8 copies of the image's visual information. This memory explosion severely limits the image resolution and batch sizes that can fit on a GPU.

The authors ask: Do we really need all those copies of the image to generate an answer?

Methods

The authors introduce FoldCache, a training-free method to compress the KV cache along the "recurrent axis." Their approach is based on a key discovery: while the model needs distinct visual states to build its understanding (the "prefill" stage), it can reuse the final visual state of a module for all its recurrent invocations during the generation (decoding) stage.

The FoldCache Workflow:

  1. Exact Prefill: The initial pass through the image and question (prefill) remains untouched. This ensures the model builds the high-quality representations it was trained to produce.
  2. Grouped Sharing: After prefill, recurrent invocations of the same module are partitioned into groups. FoldCache retains only the visual KV states from the last invocation in each group.
  3. Segmented Decoding: During text generation, when a module needs to "look" at the image, it reads from the shared "folded" visual cache. Crucially, the KV states for the question and the newly generated answer tokens remain separate for each invocation to maintain reasoning accuracy.
  4. Granularity Options:
    • FoldCache (Cycle-wise): Groups invocations within each recurrent cycle. This is the safer default.
  5. FoldCache-Lite (Module-wise): A more aggressive version that keeps only one visual copy per module across all cycles.
Figure 2. Overview of FoldCache on LoopVL. (a) During prefill, the invocations of a group write their visual KV into one shared buffer in turn, so it ends with the last invocation. (b) Cycle-wise folding (default) keeps r2, r3, r6, r7, and module-wise folding (FoldCache-Lite) keeps r6, r7. Groups never cross module boundaries. (c) At decoding, each invocation of a sample attends to its group’s visual segment, stored once per group and sample, and to its own question and answer KV, and the two results are merged by log-sum-exp.

Datasets

The researchers evaluated FoldCache using the LoopVL-1B model across five diverse open-ended VQA benchmarks:

  • ChartQA: Visual reasoning on charts.
  • MathVision: Mathematical problems involving images.
  • DocVQA: Question answering on document images.
  • InfoVQA: Reasoning over infographics.
  • TextVQA: Recognizing and reasoning about text in images.

They also used several multiple-choice benchmarks (like MMStar and AI2D) as "sanity checks" to ensure basic performance remained stable.

Key Insights and Ideas

  • The "Last-Step" Dominance: The most surprising finding is that decoding depends almost entirely on the visual KV of a module’s final invocation. Earlier invocations still need visual information, but they can "borrow" the final version without a penalty.
  • Module Boundaries Matter: You cannot share KV states between different modules (e.g., a "Vision" module and a "Language" module), even if they are both looped. Their representations live in different mathematical subspaces.
  • Prefill is Sacred: Any attempt to share KV states during the prefill stage (when the model is first "looking" at the input) causes a massive drop in accuracy. The "folding" must happen only at decoding time.
  • Orthogonality to Other Methods: FoldCache compresses the "recurrent axis." It can be stacked with existing techniques like quantization (reducing bit-precision) and token eviction (removing less important tokens) for even greater savings.

Key Results

  • Memory Savings: FoldCache approximately halves (0.52x) the prefix KV memory. The "Lite" version is even more efficient, reducing it to 0.27x.
  • Accuracy Preservation: On the five main benchmarks, FoldCache matched the accuracy of a full, uncompressed cache within a tiny margin of 0.02 points.
  • Scaling Batch Size: Because it frees up so much memory, FoldCache allows for significantly larger batch sizes on the same hardware—up to 1.7x for standard FoldCache and 2.6x for FoldCache-Lite.
  • Efficiency: When combined with 4-bit quantization (INT4), FoldCache can reduce the memory footprint to just 10-17% of the original size while remaining highly accurate.
Figure 1. Folding the visual KV of a looped VLM. LoopVL runs its two modules eight times, (L L L H) × 2, and each invocation stores its own visual KV. After an exact prefill, FoldCache keeps only the visual KV of the last invocation of each module in each cycle. The other invocations of that module read this copy (cycle 1 shown, where r0 and r1 read r2) but never the copy of another module (crossed arrow), and question KV stays separate for every invocation. Left: prefix KV memory on ChartQA and average accuracy over five benchmarks. Last-step sharing from Ouro [75] serves every invocation from the final one, across modules and for question tokens, and loses 24.6 points, whereas FoldCache stores 52% of the FullKV prefix cache at the same accuracy.

Limitations and Future Work

  • Text-Dense Sensitivity: The more aggressive "Lite" version showed some sensitivity on text-dense images (like those in TextVQA), losing about 0.9 points. For these tasks, the cycle-wise default is preferred.
  • Model Specificity: The study focused on LoopVL. While the principles likely apply to other looped architectures (like Ouro or RecursiveVLM), they haven't been tested there yet.
  • Speculative Decoding: The authors suggest that since the model relies so heavily on the "last" states, future work could explore "self-speculative decoding," using earlier (cheaper) states to draft answers that are then verified by the final states.

Summary updated

Abstract

Looped vision-language models (VLMs) deepen visual reasoning by repeatedly applying weight-tied modules. However, their decoding memory grows with recurrent depth and limits the image size and batch size that fit in a fixed memory budget. Each recurrent step produces its own key-value (KV) states, so the cache holds several copies of the same image. We find that, after an exact prefill, earlier steps still need access to visual information but can reuse the final visual KV within the same module. We propose FoldCache, which compresses visual KV along the recurrent axis while preserving the full prefill computation. FoldCache partitions recurrent steps within each module into groups and retains the final visual KV of each group for its members to share during decoding. Sharing stays within module boundaries, while question and answer KV remain separate for each step. The grouping granularity controls the degree of compression, and folding can be combined with token eviction and quantization to further reduce memory use. On LoopVL, FoldCache approximately halves prefix KV memory while matching full-cache accuracy on five open-ended benchmarks. These memory savings support batch sizes up to 1.7 times those of a full cache under the same GPU memory budget, and up to 2.6 times with the more aggressive FoldCache-Lite.
BibTeX
@misc{zhou2026fold,
  author = {Kun Zhou and Wenmian Yang and Zhensheng Wang and Yanzeng Li and Weijia Jia},
  title = {{Fold the Loop: Decoding-Time KV Memory for Looped Vision-Language Models}},
  year = {2026},
  eprint = {2611.25369},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25369},
}
Figure