TL;DR
Reviso is an agentic video captioning framework that moves away from treating all video frames as equally important. By building a hierarchical "Context–Focus–Points" plan and using specialized tools like OCR and temporal zooming, it learns to actively gather the most relevant visual evidence to produce highly accurate, context-aware captions.
What Problem Does This Paper Address?
Multimodal Large Language Models (MLLMs) have become quite good at describing videos, but they suffer from a "forest for the trees" problem. They often hallucinate or overemphasize trivial details while missing the core events or essential context.
The fundamental issue is that existing models typically process videos through passive, uniform sampling. However, not all visual evidence deserves equal attention. A cooking tutorial requires close-up focus on hand actions, while a sports highlight might require reading a tiny scoreboard. If a model only sees a low-resolution, low-frame-rate overview, it lacks the "active perception" necessary to zoom in on what actually matters for a high-quality description.
Methods
The authors propose Reviso, a framework that treats video captioning as an active, iterative process.
1. The CFP Observation Plan
Instead of jumping straight to a caption, Reviso first generates a hierarchical Context–Focus–Points (CFP) plan based on a low-FPS global preview:
- Context: Identifies the setting (e.g., "Skincare advertisement").
- Focus: Prioritizes what to look for (e.g., "Product name and label text").
- Points: Defines specific targets and tools (e.g., "Use OCR on the bottle at 0:05").
2. Agentic Observation
Guided by the CFP plan, the model acts as an agent. It can invoke tools to gather "local evidence," such as:
- High-FPS Revisit: Looking at a specific segment with more temporal detail.
- Crop & Zoom: Focusing on a specific spatial region.
- OCR: Extracting text from the scene.
3. Training Strategy
To teach the model how to observe, the authors developed a two-stage pipeline:
- Supervised Fine-Tuning (SFT): The model is trained on "verified teacher trajectories." A stronger model (like Gemini 3.1 Pro) acts as a teacher to generate and verify successful observation paths.
- Reinforcement Learning (GRPO): The model is further refined using Group Relative Policy Optimization. They introduce three specific rewards:
- Key-Fact Reward (KFR): Ensures the final caption covers essential facts.
- Plan Alignment Reward (PAR): Rewards the model for creating a plan that matches the actual information needs of the video.
- Unproductive Tool-Call Penalty (UTCP): A clever penalty that discourages "expensive" tool use (like extra frame processing) unless it actually leads to a better caption.
Datasets
Reviso was trained on a mix of 10,000 SFT trajectories and 20,000 RL samples drawn from LLaVA-Video-178K, Vript, Ego4D, and YouTube.
For evaluation, the authors tested across four major captioning benchmarks: DREAM-1K, MCTS-VCB, CaReBench, and VDC. They also evaluated downstream performance on Video Question Answering (VideoQA) using LVBench and MMVU.
Key Insights and Ideas
- Priorities Matter: The paper highlights that observation priorities are highly domain-specific. As shown in Figure 3, a robotics video needs action-order focus, while a scenic video needs atmosphere and lighting details.
- Active vs. Passive Scaling: Simply increasing the frame rate (FPS) for the whole video doesn't always improve quality and can even degrade performance for some models. Reviso proves that targeted high-FPS revisits are more efficient and effective than uniform dense sampling.
- The "Aha" Moment in Tool Use: The qualitative results (Figure 6) show Reviso correctly identifying a basketball score by using OCR, whereas the base model (Qwen3.5) hallucinated that the game had ended because it couldn't see the scoreboard clearly.
Key Results
- State-of-the-Art Performance: Reviso outperformed all evaluated open-source baselines. On the MCTS-VCB benchmark, it achieved a 15.4-point F1 improvement over its Qwen3.5-9B backbone.
- Better for Downstream Tasks: Captions generated by Reviso led to a 4.9% accuracy boost on LVBench for VideoQA, suggesting the captions are more information-dense and factually grounded.
- Efficiency for Long Videos: The benefits of tool-assisted observation were most pronounced in longer videos (20+ seconds), where uniform sampling is most likely to miss fleeting but critical details.
Limitations and Future Work
- Tool Dependency: The current framework relies on a predefined set of tools (Zoom, OCR, etc.). Expanding this to more diverse tools (e.g., audio analysis or external knowledge retrieval) could further improve performance.
- Computational Overhead: While the UTCP reward helps regulate costs, agentic observation (multiple rounds of inference) is inherently more computationally expensive than single-pass captioning.
- Teacher Reliance: The training pipeline depends heavily on high-quality trajectories from proprietary models like Gemini. Developing self-correcting mechanisms that don't require a "teacher" would be a significant next step.
Summary updated