TL;DR
LAVE (Latent Visual Evidence-Enhanced Planning) is a training-free framework that allows video-understanding agents to retain and reuse "pre-verbal" latent visual features from past tool calls. By bypassing the traditional text-only communication bottleneck between an agent's planner and its visual tools, LAVE significantly boosts multi-step planning accuracy and temporal localization without requiring model retraining or costly frame replays.
What Problem Does This Paper Address?
Modern AI agents handle long-video understanding by acting like investigators: a central Planner iteratively invokes visual Tools (like skimming or focusing on specific timestamps) to scan the video. However, these systems suffer from a subtle but severe bottleneck: Tool-Planner communication is typically text-only.
When a visual tool inspects a video segment, it compresses its findings into a short textual summary (e.g., "The player celebrates"). This verbalization is inherently lossy. Rich visual details—such as jersey numbers, specific gestures, or background actions—are completely discarded if they aren't explicitly written down. If the Planner later realizes it needs those omitted details, it must re-download and re-process the raw video frames, leading to massive computational overhead and high latency. The authors term this limitation the Tool observation bottleneck.
Methods
LAVE solves this bottleneck by introducing a dual-channel observation interface that operates entirely at inference time without any additional training:
- Visible Channel: Preserves the standard textual trajectory generated by the tool.
- Latent Channel: Captures and stores intermediate, pre-verbal visual activations (hidden states) from the Tool VLM's forward pass, anchoring them to their exact source timestamps and spatial locations.
Key Mechanisms:
- Pre-verbal Evidence Acquisition: LAVE extracts key-value pairs from a specific intermediate layer (Layer 19) of the Tool VLM before the text generation phase occurs.
- Entropy-Constrained Frame-Time Routing: When the Planner formulates its next thought, LAVE evaluates the utility of stored latent states by measuring their relevance to the current query minus their redundancy with the existing text. It uses Shannon entropy to dynamically adjust its temporal search window: if the agent is highly certain, it focuses narrowly on a few frames; if it is uncertain, it broadens its retrieval across multiple plausible moments.
- Bounded Residual Injection: Instead of feeding raw pixels back to the Planner, LAVE injects the retrieved latent updates as bounded residual modifications directly into the Planner's hidden states at the exact tokens corresponding to the timestamped text lines.
Datasets
The authors evaluated LAVE across three challenging video understanding benchmarks, focusing entirely on zero-shot performance without subtitles:
- Video-MME: Evaluates multi-choice question answering across short, medium, and long video durations.
- LongVideoBench: Tests long-context video-language understanding and evaluation.
- CG-Bench (Clue-grounded Benchmark): Specifically measures an agent's ability to locate precise temporal clues within long videos.
Key Insights and Ideas
- Latent Space as a "Free" Memory Bank: Intermediate activations within Vision-Language Models (VLMs) function as highly dense, unverbalized context stores. Reusing them bypasses the need to re-encode raw pixels.
- Complementarity-Driven Retrieval: LAVE doesn't just retrieve what is relevant; it explicitly penalizes features that are already well-described in the text channel. This ensures the latent channel only provides novel, complementary information.
- Preserving Model Stability via Bounded Updates: Injecting raw hidden states can easily derail an LLM's reasoning. By scaling updates using Root-Mean-Square (RMS) matching and applying a strict global bound ($\rho_{max}$), LAVE safely steers the Planner's attention without corrupting its base capabilities.
Key Results
LAVE was integrated into the state-of-the-art agent framework VideoSeek using both Qwen3.5-9B and GLM-4.6V-Flash-9B backbones, yielding substantial improvements:
- Performance Boost: On Video-MME, LAVE achieved a 62.77% overall score, outperforming the strong VideoSeek baseline by 3.76 points and outperforming the base VLM by 14.27 points.
- Frame Efficiency: LAVE achieved these gains while maintaining a nearly identical (or slightly lower) frame budget (60.66 frames vs. 60.70 frames for the baseline), proving it successfully reuses past computation rather than just looking at more data.
- Superior Long-Video Handling: The largest performance gains occurred in long videos (+4.77 points), where multi-step tool interactions are most frequent.
- Pinpoint Accuracy: On CG-Bench, LAVE improved exact clue localization accuracy (Hit@0) by 7.15 points, demonstrating that latent memory prevents agents from drifting away from relevant video segments.
Limitations and Future Work
- Open-Weights Dependency: Because LAVE relies on hooking into intermediate layer activations and modifying hidden states, it cannot be applied to closed, black-box commercial APIs (like standard GPT-4o or Gemini endpoints) without provider-side access.
- Formatting Constraints: The framework requires the visual tools to output structured, timestamped text lines so it can accurately anchor the latent injections.
- Overhead: While it avoids expensive pixel re-processing, maintaining a latent memory bank introduces unquantified memory and retrieval latency overhead during long inference chains.
Summary updated