VideoNexus: Video Perception Beyond the Reasoning Context

Yancheng Long, Xuanyu Zheng, Wei Chen, Luoyuan Zhang, Haoyang Xu, Bin Wen, Tingting GaoarXiv 2026 · arXiv

TL;DR

VideoNexus is a training-free framework that solves the "context burden" in long-video understanding by decoupling dense perception from reasoning. By delegating video processing to independent sub-agents that return compact, temporally grounded text "Findings" and minimal "evidence cards," it achieves state-of-the-art accuracy on benchmarks like LVBench while slashing API costs by over 50%.

What Problem Does This Paper Address?

As Large Multimodal Models (LMMs) tackle longer videos, they hit a fundamental scaling wall: context bloat. A one-hour video sampled at just one frame per second generates hundreds of thousands of visual tokens.

In traditional "coupled" agent designs, when a reasoning model (the Planner) asks to see a clip, those raw pixels are injected directly into its reasoning context. As the agent iterates—looking at different parts of the video to verify details—the context fills up with redundant visual data. This "pixels pile up" effect (illustrated in Figure 1a) burdens the model’s reasoning capacity and drives up inference costs. Existing workarounds, like per-frame text descriptions, often still scale linearly with video length, failing to truly break the link between perception density and context growth.

Figure 1. Coupled versus decoupled multimodal context. Each row shows the reasoning context at one turn. (a) Retrieved video and audio accumulate alongside reasoning. (b) VideoNexus returns compact, temporally grounded Findings and a small evidence card from independent perception calls. Colors distinguish video (sea green), audio (gold), reasoning (blue), and findings (coral).

Methods

VideoNexus introduces a decoupled multimodal context (Figure 1b). The Planner never actually "sees" the video stream. Instead, it directs a set of ephemeral perception sub-agents through a structured interface:

  1. The Planner: Formulates local tasks, selects time intervals and sampling rates, and integrates evidence to reach a final answer.
  2. Overview Sub-agent: Performs a globally sparse scan (at most 1 fps) to identify up to eight candidate intervals relevant to the query.
  3. Inspect Sub-agent: Performs a targeted, dense scan of a specific interval. It returns a Finding: a JSON object containing a status (e.g., "ok", "absent"), a one-sentence conclusion, and specific claims anchored to absolute timestamps.
  4. Evidence Cards: To allow the Planner to verify visual claims without ingesting full clips, the Inspect tool returns a single "contact sheet" containing up to six key frames (Figure 2). This provides high-fidelity visual anchors at a fraction of the token cost.

The framework is entirely training-free and runs on stock agent runtimes using the Model Context Protocol (MCP).

Figure 2. VideoNexus on one recorded LVBench question. Top: the 61-minute source video, candidate intervals (coral), and the selected 30-second window. Middle: independent perception contexts, with each cell representing 100 tokens. Bottom: an Overview report, an Inspect Finding, and a six-frame evidence card on the source timeline. Perception consumes 162,824 tokens; the planner receives 982 finding tokens and one card.

Datasets

The authors evaluated VideoNexus on two primary benchmarks:

  • LVBench: A challenging benchmark featuring 1,549 multiple-choice questions on hour-scale videos across categories like reasoning, summarization, and event understanding.
  • Video-GAIA: A set of 271 open-ended questions that require both video evidence and external web searching.

Key Insights and Ideas

  • Breaking the Linear Scaling: The core "aha moment" is that the size of a perception report should depend on the evidence found, not the duration inspected. As shown in Figure 4 of the paper, VideoNexus's "Inspect" tool shows very weak output growth even as it processes more frames, whereas traditional scanning tools grow linearly.
  • Temporal Grounding as a Common Language: By forcing all sub-agents (ASR, Overview, Inspect) to report evidence on an absolute source timeline, the Planner can easily cross-reference visual findings with audio transcripts.
  • Speech is a Cheap Anchor: The authors found that using ASR (the Transcribe tool) as a first pass significantly reduces the need for expensive visual scans, acting as a "cheap localization cue."
  • Uncertainty as a Signal: The "Finding" protocol includes an explicit "uncertainty" status. If a sub-agent can't confirm a detail due to sparse sampling, it tells the Planner, which can then trigger a higher-density re-inspection.

Key Results

Using Gemini 3.8 Flash as the backbone, VideoNexus demonstrated impressive gains in both performance and efficiency:

  • Accuracy: Achieved 88.6% accuracy on the full LVBench, outperforming static ingestion (85.0%) and Gemini’s own built-in agentic mode (87.5%).
  • Cost Efficiency: It achieved these results at only 44% of the API cost of the built-in agentic mode.
  • Context Savings: In ablation studies, decoupling perception reduced the number of tokens the Planner had to process by 32% while actually increasing accuracy.
  • Cross-Model Success: When swapped to a Kimi K3 backbone on Video-GAIA, VideoNexus improved over the official ReAct harness by 8.5 points.
Cost vs. Accuracy Comparison

Limitations and Future Work

  • Latency: Each sub-agent call is an independent API request, which can increase the wall-clock time compared to a single long-context call.
  • Spatial Resolution: While the framework handles temporal density well, very small objects might still be missed if they aren't captured in the "evidence card" or if the sub-agent's internal resolution is too low.
  • Adaptive Strategies: The current "skill" (policy) for the Planner is relatively simple (~10 sentences). Future work could explore more sophisticated, learned policies for deciding when to increase sampling density versus when to move on.
  • Text Priors: The authors acknowledge a "text prior" in benchmarks; a blind model (no video) still scores 40.5% on LVBench, suggesting that some questions can be "guessed" from the options alone.

Summary updated

Abstract

Understanding long videos requires locating evidence, resolving visual details, and integrating observations into an answer. Even when relevant evidence is present, performing these operations within long, dense visual sequences burdens both perception and reasoning. Video agents enable selective inspection, but returning frames to the planner still accumulates visual information in its reasoning context. Delegating perception reduces this burden, yet sparse sampling can miss decisive details, and per-frame descriptions can preserve context growth. We introduce VideoNexus, a training-free framework that separates detailed video perception from evidence-driven planning and reasoning. The planner controls where to inspect, what to resolve, and how densely to sample. Perception sub-agents process native audio-video in independent contexts and return compact, temporally grounded evidence for iterative search and verification. Using Gemini 3.8 Flash for both planning and perception, VideoNexus achieves 88.6% accuracy on full LVBench, exceeding same-backbone static ingestion and built-in agentic processing by 3.6 and 1.1 points, respectively, at 44% of the latter's measured API cost. A paired LVBench subset ablation shows lower planner input and higher accuracy. With Kimi K3 as the shared backbone, VideoNexus improves over the official Video-GAIA ReAct harness by 8.5 points in our evaluation. These results support compact evidence exchange for combining detailed observation with efficient planning and reasoning.
BibTeX
@misc{long2026videonexus,
  author = {Yancheng Long and Xuanyu Zheng and Wei Chen and Luoyuan Zhang and Haoyang Xu and Bin Wen and Tingting Gao},
  title = {{VideoNexus: Video Perception Beyond the Reasoning Context}},
  year = {2026},
  eprint = {2611.21392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21392},
}
Figure