TL;DR
Looped Vision-Language Models (VLMs) achieve deeper reasoning by recycling the same neural modules multiple times, but this creates a massive memory bottleneck: the Key-Value (KV) cache grows with every "loop." FoldCache solves this by identifying that visual representations are redundant across these loops during decoding; by "folding" these states so that multiple recurrent steps share a single set of visual KV entries, FoldCache halves memory usage with virtually zero loss in accuracy.
What Problem Does This Paper Address?
Modern VLMs are increasingly using "looped" or recurrent architectures. Instead of just passing an image through a long sequence of unique layers, these models pass representations through the same weight-tied modules multiple times. This "recurrent depth" allows the model to "think longer" about an image without increasing the number of parameters.
However, there’s a catch: the KV Cache. During the decoding phase (when the model generates text token by token), it stores previously computed Key and Value states to avoid redundant calculations. In a looped VLM, every single pass through a module generates a new set of KV states for the same image tokens. If a model loops 8 times, the cache holds 8 copies of the image's visual information. This memory explosion severely limits the image resolution and batch sizes that can fit on a GPU.
The authors ask: Do we really need all those copies of the image to generate an answer?
Methods
The authors introduce FoldCache, a training-free method to compress the KV cache along the "recurrent axis." Their approach is based on a key discovery: while the model needs distinct visual states to build its understanding (the "prefill" stage), it can reuse the final visual state of a module for all its recurrent invocations during the generation (decoding) stage.
The FoldCache Workflow:
- Exact Prefill: The initial pass through the image and question (prefill) remains untouched. This ensures the model builds the high-quality representations it was trained to produce.
- Grouped Sharing: After prefill, recurrent invocations of the same module are partitioned into groups. FoldCache retains only the visual KV states from the last invocation in each group.
- Segmented Decoding: During text generation, when a module needs to "look" at the image, it reads from the shared "folded" visual cache. Crucially, the KV states for the question and the newly generated answer tokens remain separate for each invocation to maintain reasoning accuracy.
- Granularity Options:
- FoldCache (Cycle-wise): Groups invocations within each recurrent cycle. This is the safer default.
- FoldCache-Lite (Module-wise): A more aggressive version that keeps only one visual copy per module across all cycles.
Datasets
The researchers evaluated FoldCache using the LoopVL-1B model across five diverse open-ended VQA benchmarks:
- ChartQA: Visual reasoning on charts.
- MathVision: Mathematical problems involving images.
- DocVQA: Question answering on document images.
- InfoVQA: Reasoning over infographics.
- TextVQA: Recognizing and reasoning about text in images.
They also used several multiple-choice benchmarks (like MMStar and AI2D) as "sanity checks" to ensure basic performance remained stable.
Key Insights and Ideas
- The "Last-Step" Dominance: The most surprising finding is that decoding depends almost entirely on the visual KV of a module’s final invocation. Earlier invocations still need visual information, but they can "borrow" the final version without a penalty.
- Module Boundaries Matter: You cannot share KV states between different modules (e.g., a "Vision" module and a "Language" module), even if they are both looped. Their representations live in different mathematical subspaces.
- Prefill is Sacred: Any attempt to share KV states during the prefill stage (when the model is first "looking" at the input) causes a massive drop in accuracy. The "folding" must happen only at decoding time.
- Orthogonality to Other Methods: FoldCache compresses the "recurrent axis." It can be stacked with existing techniques like quantization (reducing bit-precision) and token eviction (removing less important tokens) for even greater savings.
Key Results
- Memory Savings: FoldCache approximately halves (0.52x) the prefix KV memory. The "Lite" version is even more efficient, reducing it to 0.27x.
- Accuracy Preservation: On the five main benchmarks, FoldCache matched the accuracy of a full, uncompressed cache within a tiny margin of 0.02 points.
- Scaling Batch Size: Because it frees up so much memory, FoldCache allows for significantly larger batch sizes on the same hardware—up to 1.7x for standard FoldCache and 2.6x for FoldCache-Lite.
- Efficiency: When combined with 4-bit quantization (INT4), FoldCache can reduce the memory footprint to just 10-17% of the original size while remaining highly accurate.
Limitations and Future Work
- Text-Dense Sensitivity: The more aggressive "Lite" version showed some sensitivity on text-dense images (like those in TextVQA), losing about 0.9 points. For these tasks, the cycle-wise default is preferred.
- Model Specificity: The study focused on LoopVL. While the principles likely apply to other looped architectures (like Ouro or RecursiveVLM), they haven't been tested there yet.
- Speculative Decoding: The authors suggest that since the model relies so heavily on the "last" states, future work could explore "self-speculative decoding," using earlier (cheaper) states to draft answers that are then verified by the final states.
Summary updated