CanopyKV: Remember beyond the Crop for Fast Active Visual Reasoning

Lei Yang, Dayan Wu, Yiliang Zhu, Xinze Liu, Hengjie Zhu, Zihao Zhang, Ding Wang, Peng Fu, Zheng LinarXiv 2026 · arXiv

TL;DR

CanopyKV is a training-free KV cache compression framework designed specifically for "active visual reasoning," where models iteratively zoom into images to find answers. By using spatial "crop lineage" to retain evidence outside of new focal points, it achieves state-of-the-art accuracy while reducing memory usage to 25% and accelerating inference by up to 1.64×.

What Problem Does This Paper Address?

Modern multimodal models often solve complex visual tasks through active reasoning: they look at an image, decide which parts are interesting, "crop" into those regions for a closer look, and repeat. While effective, this process creates a massive Key-Value (KV) cache—the model's "working memory"—which grows with every new crop and reasoning step.

Existing KV compression methods (like those based on attention scores or recency) struggle here because they don't understand the spatial relationship between views. When a model zooms into a small detail (a "child" crop), standard pruning might discard the surrounding context from the "parent" image. However, that surrounding context is often vital for future steps or for maintaining a "big picture" understanding. As shown in the paper, evidence outside a new crop is significantly harder for a model to recover if it's been pruned, leading to reasoning failures or expensive re-searching.

Local replacement vs. persistent complementarity. Standard pruning often loses context outside the current crop, while CanopyKV retains it.

Methods

CanopyKV introduces a training-free approach that leverages the structural metadata already provided by visual tools (like crop coordinates) to manage memory efficiently. Its core components include:

  • Lineage-Guided Spatial Retention: Instead of using noisy attention scores, CanopyKV uses "crop lineage." When a new crop is acquired, the system identifies which parts of the parent image are not covered by the new view and prioritizes keeping those "complementary" tokens.
  • Source-Aware Quotas: It balances the memory budget between the original root image, subsequent crops, and the reasoning text itself, ensuring no single source dominates the cache.
  • Stable Continuation: Unlike methods that re-rank and re-prune the entire history at every step, CanopyKV "retires" a packet of information once. Once selected, that history remains fixed, which drastically reduces the computational overhead of cache management.
  • Native KV Execution: The framework operates directly on native KV tensors, allowing it to benefit from optimized kernels like FlashAttention without needing extra model forward passes for scoring.
Figure 2. CANOPYKV retains complementary evidence in compact KV. (a) Source quotas and crop lineage favor context outside the incoming crop. (b) One-time retirement protects continuation state and fixes selected history across branch visits. (c) Incremental prefill is followed by visual pruning and KV compaction, preserving current text and logical positions.

Datasets

The authors evaluated CanopyKV across several challenging benchmarks for active visual reasoning:

  • VisualProbe: Focuses on iterative tool use for fine-grained reasoning.
  • V*: A benchmark for guided visual search.
  • HR-Bench (4K & 8K): Tests high-resolution visual perception.
  • MME-RealWorld Lite (MME-Lite): A comprehensive suite for real-world multimodal reasoning.

Key Insights and Ideas

  • Persistent Complementarity: The "aha moment" of the paper is the realization that a new crop doesn't make the parent image redundant; it only refreshes a local region. The evidence outside the new crop's footprint is the most vulnerable to pruning and the most important to protect.
  • Metadata is a Stronger Signal than Attention: In active reasoning, the geometric relationship between crops is a more reliable indicator of "what to remember" than the model's internal attention weights, which can be distracted by transient reasoning steps.
  • The Cost of Re-searching: Pruning the wrong tokens doesn't just hurt accuracy; it increases the "search effort." A model with a poorly pruned cache might take more turns to find an answer, offsetting any speed gains from having a smaller cache. CanopyKV minimizes this by keeping the right evidence available.
Figure 3. Parent evidence outside the new crop is harder to replace. After removing matched amounts of parent KV inside or outside the incoming crop, we supply the full crop and read out the corresponding facts. The horizontal axes show KV retention within each target fact region.

Key Results

CanopyKV consistently outperformed both "Dense" models (which keep everything) and state-of-the-art compression baselines:

  • Accuracy: On MME-Lite, it improved accuracy by 10.5 percentage points over the Dense model while retaining only ~25% of the KV cache.
  • Efficiency: It achieved an end-to-end reasoning speedup of 1.64× on MME-Lite. Across all benchmarks, it was faster than every compared baseline, including VL-Cache and MixKV.
  • Search Depth: By retaining better evidence, CanopyKV often required fewer reasoning rounds than other compression methods, which frequently got "lost" and required extra crops to recover missing information.

Limitations and Future Work

  • Tool Dependency: CanopyKV relies on explicit metadata (coordinates) from visual tools. It might require adaptation for models that perform "cropping" implicitly or through less structured interfaces.
  • Hyperparameter Sensitivity: The "source quotas" (how much to weight text vs. images) are currently fixed defaults. Future work could explore making these allocations adaptive based on the complexity of the question.
  • Scaling to Video: While designed for static images with iterative crops, the concept of "spatial lineage" could potentially be extended to temporal lineage in long-form video reasoning.

Summary updated

Abstract

Active visual reasoning enables multimodal models to answer challenging visual questions by gathering fine-grained evidence through iterative image inspection and tool use. However, accumulating crops and reasoning tokens expands the key-value (KV) cache, making compression important for reducing memory and inference costs. Existing KV compression methods typically select historical tokens based on attention, recency, or redundancy. These criteria do not explicitly distinguish locally replaced evidence from complementary information needed for future exploration: each crop refreshes a local region, while earlier views may still support subsequent decisions. Discarding such evidence can impair accuracy or require additional search, while scoring and cache reorganization can offset the computational savings from compression. We introduce CanopyKV, a training-free KV compression framework that uses parent-child crop relationships to retain complementary evidence, keeps retained history stable, and protects the state needed for ongoing reasoning. Tool metadata provides retention signals without additional attention scoring, while compact KV storage reduces subsequent attention costs. Across VisualProbe, V*, HR-Bench, and MME-Lite, CanopyKV achieves state-of-the-art answer accuracy and end-to-end efficiency among the evaluated KV compression methods. On MME-Lite, it retains approximately one-quarter of Dense's total KV while improving accuracy by 10.5 percentage points and accelerating complete reasoning by $1.64\times$.
BibTeX
@misc{yang2026canopykv,
  author = {Lei Yang and Dayan Wu and Yiliang Zhu and Xinze Liu and Hengjie Zhu and Zihao Zhang and Ding Wang and Peng Fu and Zheng Lin},
  title = {{CanopyKV: Remember beyond the Crop for Fast Active Visual Reasoning}},
  year = {2026},
  eprint = {2611.24373},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24373},
}
Figure