Keep the Reasoning, Read the Image: On-Demand Visual KV for Multi-Turn Reasoning VLMs

Ziyi Wang, Xiyue Zhang, Bing Hu, Guangmin Song, Chenyi Zhang, Li WangarXiv 2026 · arXiv

TL;DR

Reasoning Vision-Language Models (VLMs) often "forget" the images they are looking at during long conversations because their internal memory (KV cache) gets crowded out by their own lengthy chains of thought. ShelfKV solves this by storing the entire image on a "shelf" in CPU memory and dynamically "reading" only the most relevant parts into the GPU on demand, preventing visual information from being permanently deleted.

What Problem Does This Paper Address?

Modern "reasoning" VLMs (like Qwen3-VL-Thinking) are designed to "think" before they speak, generating thousands of internal tokens to solve complex math or chart problems. This internal reasoning is stored in a Key-Value (KV) cache to speed up processing. However, GPU memory is limited, so models use "retention policies" to prune this cache, deleting older or less important entries.

The authors discovered a critical flaw: in multi-turn conversations, the model's own reasoning tokens compete for the same memory budget as the original image. As the reasoning grows, it "crowds out" the visual data. If a user asks a follow-up question about a part of the image the model didn't use in its first round of reasoning, that information is often already gone, leading to a massive drop in accuracy.

Figure 2. Reasoning crowds out visual KV; protection beats extra shared capacity. (a) Cache composition during the first-turn output under DBTrimKV (B = 256). (b) Follow-up accuracy gains (percentage points) over the same host (61.3%); the dashed line marks the unconstrained full-cache gain.

Methods

The authors propose ShelfKV, a training-free method that separates keeping visual information from reading it.

  1. The Shelf (CPU): Instead of letting the image compete for GPU space, ShelfKV stores every visual KV entry in CPU memory (the "shelf"). These are never deleted.
  2. The Desk (GPU): The GPU cache (the "desk") maintains a fixed budget, primarily managed by a standard text retention policy.
  3. On-Demand Reading: At each step of generating an answer, ShelfKV identifies which visual parts are needed. It uses a "sketch"—a compact, low-bit version of the visual keys—to quickly score candidates.
  4. Pairwise Swap: If a visual entry on the shelf is more relevant to the current query than a text entry currently on the desk, they swap places. The text entry is "parked" (temporarily moved aside) rather than deleted, and it returns to the desk once the visual entry is no longer needed.
Figure 1. Keep the reasoning, read the image. (a) A retention policy decides what to keep while the model reasons, so a follow-up that needs a region the reasoning did not use finds it deleted. ShelfKV keeps every visual entry on a shelf in CPU memory and moves the entries the current query needs onto the desk that attention reads. (b) Under DBTrimKV, a descriptive CharXiv question loses three times as much accuracy after a round of reasoning as when asked first; reading the image on demand at matched GPU memory closes most of the gap.

Datasets

The researchers evaluated ShelfKV across four benchmarks:

  • CharXiv (Follow-ups & Conversations): A dataset focused on complex chart reasoning and descriptive follow-up questions.
  • MMMU-Pro Vision: A benchmark for multi-discipline multimodal understanding.
  • MMDU: A multi-turn, multi-image dialogue understanding benchmark.

Key Insights and Ideas

  • Protection, Not Capacity: The image isn't being evicted because the cache is too small; it's being evicted because it isn't protected. The authors found that a small "protected reserve" for visual tokens outperformed a much larger "shared budget" where text and images competed equally.
  • Dynamic Relevance: What a model needs to "see" changes turn-by-turn. A static pruning policy at the start of a conversation cannot predict what a user might ask four turns later.
  • Training-Free Integration: ShelfKV works as a "plug-in" on top of existing state-of-the-art retention policies like DBTrimKV or R-KV, requiring no additional model training.

Key Results

ShelfKV significantly closes the accuracy gap caused by cache pruning in multi-turn reasoning:

  • Massive Accuracy Gains: On CharXiv follow-up questions, ShelfKV improved the accuracy of the DBTrimKV policy by 20.3 points at the same GPU memory level.
  • Efficiency: ShelfKV with a smaller GPU budget outperformed standard policies using twice the memory.
  • Consistency: It consistently beat "fixed reserve" strategies (which set aside a permanent spot for images) because it can dynamically pull in the right parts of the image for the current question.
  • Zero-Shot Success: It maintains high performance on standalone questions while dramatically improving multi-turn performance.

Limitations and Future Work

  • Latency Trade-off: Reading from CPU memory and scoring sketches adds a small per-step computational cost, slightly slowing down decoding speed (e.g., from ~41ms to ~48ms per token in some settings).
  • Text Eviction: While ShelfKV protects the image, the underlying "host" policy still deletes text entries to stay within budget. If the conversation becomes extremely long, the model might still "forget" earlier parts of the text dialogue.
  • Prefill Only: The current implementation handles images provided at the start (prefill). Future work could extend this to handle images or videos introduced mid-conversation.

Summary updated

Abstract

Reasoning vision-language models (VLMs) think for thousands of tokens before they answer, and retention policies keep the KV cache affordable by pruning it as the reasoning grows. We find that this pruning sacrifices the image in multi-turn conversations: the image and the reasoning share one budget, so the growing reasoning crowds the visual entries out. A chart question that loses 8 points to the full cache when asked first loses 24 points when asked after a round of reasoning. Protecting a small visual reserve from eviction recovers most of this loss, while adding the same slots to the shared budget recovers considerably less. The image therefore lacks protection, not capacity. Based on this finding, we propose ShelfKV, a training-free method that works on top of existing retention policies. It keeps every visual entry in CPU memory and, at each decoding step, swaps the visual entries that the current query scores highest into the attention slots of the text entries it scores lowest. The image is never evicted, yet it occupies attention slots only while it is read. We evaluate ShelfKV on four benchmarks, three of them conversational, with two retention policies and two reasoning VLMs. In conversations, it improves the strongest existing policy, DBTrimKV, by up to 20.3 points at matched GPU memory, and still leads by 8.7 points when DBTrimKV uses twice the memory. It also consistently outperforms a fixed visual reserve of matched GPU memory in these settings, without hurting accuracy on standalone questions. These results show that the image should be protected from the reasoning but read only on demand.
BibTeX
@misc{wang2026keep,
  author = {Ziyi Wang and Xiyue Zhang and Bing Hu and Guangmin Song and Chenyi Zhang and Li Wang},
  title = {{Keep the Reasoning, Read the Image: On-Demand Visual KV for Multi-Turn Reasoning VLMs}},
  year = {2026},
  eprint = {2611.23769},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23769},
}
Figure