LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents

Zijian Wang, Junnan Zhu, Rongzhen Li, Xiao Liu, Guohui Xiang, Quan Lu, Lijia Liu, Yining Wang, Jiang Zhong, Kaiwen WeiarXiv 2026 · arXiv

TL;DR

LAVE (Latent Visual Evidence-Enhanced Planning) is a training-free framework that allows video-understanding agents to retain and reuse "pre-verbal" latent visual features from past tool calls. By bypassing the traditional text-only communication bottleneck between an agent's planner and its visual tools, LAVE significantly boosts multi-step planning accuracy and temporal localization without requiring model retraining or costly frame replays.

What Problem Does This Paper Address?

Modern AI agents handle long-video understanding by acting like investigators: a central Planner iteratively invokes visual Tools (like skimming or focusing on specific timestamps) to scan the video. However, these systems suffer from a subtle but severe bottleneck: Tool-Planner communication is typically text-only.

When a visual tool inspects a video segment, it compresses its findings into a short textual summary (e.g., "The player celebrates"). This verbalization is inherently lossy. Rich visual details—such as jersey numbers, specific gestures, or background actions—are completely discarded if they aren't explicitly written down. If the Planner later realizes it needs those omitted details, it must re-download and re-process the raw video frames, leading to massive computational overhead and high latency. The authors term this limitation the Tool observation bottleneck.

Figure 1: Comparison of video agent Tool–Planner com- munication paradigms. Text-only observations discard un- verbalized visual evidence, while raw visual inputs require additional visual inference. LAVE preserves the text trajec- tory and reuses temporally grounded latent visual evidence from completed Tool calls to improve subsequent planning.

Methods

LAVE solves this bottleneck by introducing a dual-channel observation interface that operates entirely at inference time without any additional training:

  1. Visible Channel: Preserves the standard textual trajectory generated by the tool.
  2. Latent Channel: Captures and stores intermediate, pre-verbal visual activations (hidden states) from the Tool VLM's forward pass, anchoring them to their exact source timestamps and spatial locations.
Figure 2: Overview of LAVE. Selected frames and Planner messages are processed by the Tool VLM, producing text observations and pre-verbal visual states. During Tool execution, LAVE (A) extracts latent key–value evidence and (B) stores addressed evidence with source provenance. During the next Planner prefill, LAVE (C) retrieves relevant source-time groups using the current Planner state and (D) injects bounded residuals at resolved receiver anchors to guide the next thought and Tool action.

Key Mechanisms:

  • Pre-verbal Evidence Acquisition: LAVE extracts key-value pairs from a specific intermediate layer (Layer 19) of the Tool VLM before the text generation phase occurs.
  • Entropy-Constrained Frame-Time Routing: When the Planner formulates its next thought, LAVE evaluates the utility of stored latent states by measuring their relevance to the current query minus their redundancy with the existing text. It uses Shannon entropy to dynamically adjust its temporal search window: if the agent is highly certain, it focuses narrowly on a few frames; if it is uncertain, it broadens its retrieval across multiple plausible moments.
  • Bounded Residual Injection: Instead of feeding raw pixels back to the Planner, LAVE injects the retrieved latent updates as bounded residual modifications directly into the Planner's hidden states at the exact tokens corresponding to the timestamped text lines.
Figure 3: Entropy-constrained frame-time routing. (A) Planner-state relevance and temporal entropy select source- time groups and form bounded residuals. (B) Each residual is written at the timestamp-matched receiver anchor during Planner prefill.

Datasets

The authors evaluated LAVE across three challenging video understanding benchmarks, focusing entirely on zero-shot performance without subtitles:

  • Video-MME: Evaluates multi-choice question answering across short, medium, and long video durations.
  • LongVideoBench: Tests long-context video-language understanding and evaluation.
  • CG-Bench (Clue-grounded Benchmark): Specifically measures an agent's ability to locate precise temporal clues within long videos.

Key Insights and Ideas

  • Latent Space as a "Free" Memory Bank: Intermediate activations within Vision-Language Models (VLMs) function as highly dense, unverbalized context stores. Reusing them bypasses the need to re-encode raw pixels.
  • Complementarity-Driven Retrieval: LAVE doesn't just retrieve what is relevant; it explicitly penalizes features that are already well-described in the text channel. This ensures the latent channel only provides novel, complementary information.
  • Preserving Model Stability via Bounded Updates: Injecting raw hidden states can easily derail an LLM's reasoning. By scaling updates using Root-Mean-Square (RMS) matching and applying a strict global bound ($\rho_{max}$), LAVE safely steers the Planner's attention without corrupting its base capabilities.

Key Results

LAVE was integrated into the state-of-the-art agent framework VideoSeek using both Qwen3.5-9B and GLM-4.6V-Flash-9B backbones, yielding substantial improvements:

  • Performance Boost: On Video-MME, LAVE achieved a 62.77% overall score, outperforming the strong VideoSeek baseline by 3.76 points and outperforming the base VLM by 14.27 points.
  • Frame Efficiency: LAVE achieved these gains while maintaining a nearly identical (or slightly lower) frame budget (60.66 frames vs. 60.70 frames for the baseline), proving it successfully reuses past computation rather than just looking at more data.
  • Superior Long-Video Handling: The largest performance gains occurred in long videos (+4.77 points), where multi-step tool interactions are most frequent.
  • Pinpoint Accuracy: On CG-Bench, LAVE improved exact clue localization accuracy (Hit@0) by 7.15 points, demonstrating that latent memory prevents agents from drifting away from relevant video segments.

Limitations and Future Work

  • Open-Weights Dependency: Because LAVE relies on hooking into intermediate layer activations and modifying hidden states, it cannot be applied to closed, black-box commercial APIs (like standard GPT-4o or Gemini endpoints) without provider-side access.
  • Formatting Constraints: The framework requires the visual tools to output structured, timestamped text lines so it can accurately anchor the latent injections.
  • Overhead: While it avoids expensive pixel re-processing, maintaining a latent memory bank introduces unquantified memory and retrieval latency overhead during long inference chains.

Summary updated

Abstract

Long-video understanding requires models to efficiently acquire and reuse sparse visual evidence from long and redundant video streams. Recent video tool-use agents address this challenge by iteratively invoking visual Tools at different temporal scales, but their Tool-Planner communication typically relies on textual observations. Such text-only interfaces provide lossy summaries of Tool computations, causing previously computed visual evidence not verbalized to be discarded and unavailable for subsequent planning. We identify this limitation as the Tool observation bottleneck and propose Latent Visual Evidence-Enhanced Planning (LAVE), a training-free framework for reusing latent visual evidence from completed Tool calls. LAVE introduces a dual-channel observation interface: the visible channel preserves the original textual trajectory, while the latent channel stores pre-verbal visual updates with their Tool roles, source-frame timestamps, and visual locations. During planning, LAVE retrieves evidence relevant to the current Planner state but not covered by textual observations, and integrates it through bounded timestamp-aligned latent updates with entropy-constrained frame-time routing. This enables video agents to reuse existing visual computation without additional training, frame replay, or modifications to the original orchestration. Extensive experiments on Video-MME, LongVideoBench, and CG-Bench show that LAVE consistently improves video tool-use agents across backbones. Under a comparable frame budget, LAVE improves the Video-MME overall score by 3.76 points over the strongest baseline, demonstrating the effectiveness of latent visual evidence reuse for multi-step video-agent planning.
BibTeX
@misc{wang2026lave,
  author = {Zijian Wang and Junnan Zhu and Rongzhen Li and Xiao Liu and Guohui Xiang and Quan Lu and Lijia Liu and Yining Wang and Jiang Zhong and Kaiwen Wei},
  title = {{LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents}},
  year = {2026},
  eprint = {2608.07585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.07585},
}
Figure