Reviso: Not All Visual Evidence Deserves Equal Attention in Video Captioning

Tengfei Liu, Yang Shi, Yuran Wang, Yuqi Tang, Yizhuo Jia, Zhuoran Zhang, Qixun Wang, Shujie Li, Xuanyu Zhu, Xinyu Liu, Minghui Guo, Shixuan Liu, Haotian Wang, Yuanxing Zhang, Pengfei WanarXiv 2026 · arXiv

TL;DR

Reviso is an agentic video captioning framework that moves away from treating all video frames as equally important. By building a hierarchical "Context–Focus–Points" plan and using specialized tools like OCR and temporal zooming, it learns to actively gather the most relevant visual evidence to produce highly accurate, context-aware captions.

What Problem Does This Paper Address?

Multimodal Large Language Models (MLLMs) have become quite good at describing videos, but they suffer from a "forest for the trees" problem. They often hallucinate or overemphasize trivial details while missing the core events or essential context.

The fundamental issue is that existing models typically process videos through passive, uniform sampling. However, not all visual evidence deserves equal attention. A cooking tutorial requires close-up focus on hand actions, while a sports highlight might require reading a tiny scoreboard. If a model only sees a low-resolution, low-frame-rate overview, it lacks the "active perception" necessary to zoom in on what actually matters for a high-quality description.

Methods

The authors propose Reviso, a framework that treats video captioning as an active, iterative process.

1. The CFP Observation Plan

Instead of jumping straight to a caption, Reviso first generates a hierarchical Context–Focus–Points (CFP) plan based on a low-FPS global preview:

  • Context: Identifies the setting (e.g., "Skincare advertisement").
  • Focus: Prioritizes what to look for (e.g., "Product name and label text").
  • Points: Defines specific targets and tools (e.g., "Use OCR on the bottle at 0:05").
The Reviso framework uses a hierarchical plan to guide tool-assisted observation.

2. Agentic Observation

Guided by the CFP plan, the model acts as an agent. It can invoke tools to gather "local evidence," such as:

  • High-FPS Revisit: Looking at a specific segment with more temporal detail.
  • Crop & Zoom: Focusing on a specific spatial region.
  • OCR: Extracting text from the scene.

3. Training Strategy

To teach the model how to observe, the authors developed a two-stage pipeline:

  • Supervised Fine-Tuning (SFT): The model is trained on "verified teacher trajectories." A stronger model (like Gemini 3.1 Pro) acts as a teacher to generate and verify successful observation paths.
  • Reinforcement Learning (GRPO): The model is further refined using Group Relative Policy Optimization. They introduce three specific rewards:
  • Key-Fact Reward (KFR): Ensures the final caption covers essential facts.
  • Plan Alignment Reward (PAR): Rewards the model for creating a plan that matches the actual information needs of the video.
  • Unproductive Tool-Call Penalty (UTCP): A clever penalty that discourages "expensive" tool use (like extra frame processing) unless it actually leads to a better caption.

Datasets

Reviso was trained on a mix of 10,000 SFT trajectories and 20,000 RL samples drawn from LLaVA-Video-178K, Vript, Ego4D, and YouTube.

For evaluation, the authors tested across four major captioning benchmarks: DREAM-1K, MCTS-VCB, CaReBench, and VDC. They also evaluated downstream performance on Video Question Answering (VideoQA) using LVBench and MMVU.

Key Insights and Ideas

  • Priorities Matter: The paper highlights that observation priorities are highly domain-specific. As shown in Figure 3, a robotics video needs action-order focus, while a scenic video needs atmosphere and lighting details.
Observation priorities vary significantly across different video categories.
  • Active vs. Passive Scaling: Simply increasing the frame rate (FPS) for the whole video doesn't always improve quality and can even degrade performance for some models. Reviso proves that targeted high-FPS revisits are more efficient and effective than uniform dense sampling.
  • The "Aha" Moment in Tool Use: The qualitative results (Figure 6) show Reviso correctly identifying a basketball score by using OCR, whereas the base model (Qwen3.5) hallucinated that the game had ended because it couldn't see the scoreboard clearly.
Qualitative comparison showing Reviso using OCR to verify game state.

Key Results

  • State-of-the-Art Performance: Reviso outperformed all evaluated open-source baselines. On the MCTS-VCB benchmark, it achieved a 15.4-point F1 improvement over its Qwen3.5-9B backbone.
  • Better for Downstream Tasks: Captions generated by Reviso led to a 4.9% accuracy boost on LVBench for VideoQA, suggesting the captions are more information-dense and factually grounded.
  • Efficiency for Long Videos: The benefits of tool-assisted observation were most pronounced in longer videos (20+ seconds), where uniform sampling is most likely to miss fleeting but critical details.

Limitations and Future Work

  • Tool Dependency: The current framework relies on a predefined set of tools (Zoom, OCR, etc.). Expanding this to more diverse tools (e.g., audio analysis or external knowledge retrieval) could further improve performance.
  • Computational Overhead: While the UTCP reward helps regulate costs, agentic observation (multiple rounds of inference) is inherently more computationally expensive than single-pass captioning.
  • Teacher Reliance: The training pipeline depends heavily on high-quality trajectories from proprietary models like Gemini. Developing self-correcting mechanisms that don't require a "teacher" would be a significant next step.

Summary updated

Abstract

Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video. Motivated by this observation, we propose Reviso, an agentic video captioning framework that learns what to observe and how to gather relevant visual evidence. From a low-frame-rate global preview, Reviso constructs a hierarchical Context--Focus--Points (CFP) observation plan that identifies the video context, prioritizes key aspects, and defines concrete observation targets and corresponding tools. Guided by this plan, Reviso performs goal-directed active perception by selectively revisiting temporal segments and combining tools for fine-grained inspection. It interprets the acquired evidence, updates its plan as needed, and integrates global and local information to generate the final caption. To learn this observation strategy, we develop a verification-guided pipeline that constructs high-quality training trajectories through factual verification and teacher-guided refinement. We further train Reviso using Group Relative Policy Optimization (GRPO), incorporating a key-fact reward for caption quality, a Plan Alignment Reward (PAR) for identifying important observation targets, and an Unproductive Tool-Call Penalty (UTCP) that adjusts observation-cost penalties based on caption quality. Experiments across four video captioning benchmarks show that Reviso outperforms the evaluated open-source baselines, while its generated captions better support downstream video question answering. These findings highlight the effectiveness and potential of priority-guided active observation for video captioning.
BibTeX
@misc{liu2026reviso,
  author = {Tengfei Liu and Yang Shi and Yuran Wang and Yuqi Tang and Yizhuo Jia and Zhuoran Zhang and Qixun Wang and Shujie Li and Xuanyu Zhu and Xinyu Liu and Minghui Guo and Shixuan Liu and Haotian Wang and Yuanxing Zhang and Pengfei Wan},
  title = {{Reviso: Not All Visual Evidence Deserves Equal Attention in Video Captioning}},
  year = {2026},
  eprint = {2611.03602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03602},
}
Figure