CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models

Joowon Kim, Seungho Shin, Joonhyung Park, Eunho YangarXiv 2026 · arXiv

TL;DR

CollabVR introduces a closed-loop framework that pairs Vision-Language Models (VLMs) and Video Generation Models (VGMs) at step-level granularity to tackle complex visual reasoning tasks. By using the VLM as an adaptive planner and real-time verifier, CollabVR effectively mitigates long-horizon drift and mid-clip simulation errors, outperforming standard test-time scaling baselines at matched compute budgets.

What Problem Does This Paper Address?

Recent paradigms in AI have shifted from "Thinking with Images" to "Thinking with Video," where Video Generation Models (VGMs) are leveraged to produce a "Chain-of-Frames"—a temporally coherent video that serves as a visual reasoning artifact. However, even state-of-the-art VGMs struggle with goal-directed, multi-step tasks due to two primary failure modes:

  1. Overloaded-prompt failure: When given a long-horizon task in a single prompt, the VGM fails to decompose it, collapsing the plan into a short rollout that quickly drifts away from the goal.
  2. Execution failure: Localized errors occur mid-clip (e.g., an object passing through a wall or losing its identity). Once introduced, these errors propagate and contaminate the remaining trajectory.

While Vision-Language Models (VLMs) excel at abstract logical reasoning and planning, they lack the ability to directly simulate physical visual worlds. Conversely, VGMs possess strong short-horizon visual priors but lack long-range logical consistency. Existing methods either plan entirely upfront (which is too rigid) or critique the video post-hoc (which intervenes too late).

Figure 1: VLM as planner, VGM as simulator. A VLM is strong at reasoning but weak at visual simulation, while a VGM simulates short clips but lacks reasoning, causing long-horizon drift and mid-clip simulation errors. CollabVR couples them in a closed loop where the VLM plans progressively and diagnoses each generated clip, turning failures into correctable signals.

Methods

The authors propose VLM-VGM Collaborative Video Reasoning (CollabVR), a training-free, closed-loop framework that tightly couples a VLM and a VGM at a sub-action granularity.

Figure 3: Overall pipeline of CollabVR. A persistent VLM plans one action at a time and, after observing each generated clip, decides whether to accept, re-generate, or re-plan. Module 1 adaptively determines the step count, and Module 2 verifies each clip and folds the verifier’s diagnosis into the next action prompt to repair the failure.

The framework consists of two core modules:

  • VLM-Driven Progressive Planning (Module 1): Instead of generating a full plan upfront, the VLM adaptively determines the step count on the fly. It generates a prompt only for the immediate next sub-action, conditioned on the history of previously accepted frames.
  • VLM-VGM Collaborative Reasoning (Module 2): After the VGM generates a short clip, the VLM acts as a verifier. It provides a structured accept or reject judgment. If rejected, it generates a precise textual diagnosis and an actionable suggestion for repair. This feedback is folded directly back into the action prompt (prompt evolution), and the VGM re-samples the clip.

Datasets

The authors evaluated CollabVR on two complementary video reasoning benchmarks:

  • Gen-ViRe: A benchmark containing 72 samples across 6 categories, evaluated using a rubric-based VLM judge (Gemini 2.5 Pro) focusing on task correctness rather than raw visual quality.
  • VBVR-Bench: A synthetic visual reasoning suite featuring a 500-sample test set spanning 5 reasoning categories with controlled ground-truth references.
  • Human-Annotated Reliability Benchmark: A newly introduced dataset of human annotations across 250 clips to validate the alignment of VLM planning, verification, and prompt evolution with expert human judgment.

Key Insights and Ideas

  • Step-Level Interventions are Crucial: Catching execution errors at the moment they occur prevents them from compounding and ruining the entire video trajectory.
  • Progressive beats Pre-planning: Generating plans step-by-step allows the model to adapt to the actual visual outputs produced by the VGM, yielding a +13% relative gain over rigid upfront planning at matched compute costs.
Figure 4: Pre-planning vs. Progressive planning on Gen-ViRe with VBVR-Wan2.2 (Module 1 only). Progressive planning achieves a +13% relative gain over pre-planning at matched cost.
  • Orthogonal to Fine-Tuning: CollabVR provides substantial improvements even when applied on top of VGMs that have already been fine-tuned for reasoning data (like VBVR-Wan2.2), proving that test-time supervision stacks beneficially with training-time optimization.

Key Results

CollabVR consistently outperforms single-inference, Pass@$k$ resampling, and prior test-time scaling baselines like VideoTPO across both open-source and closed-source video generation models.

Performance-cost trade-off on Gen-ViRe comparing CollabVR against baselines using VBVR-Wan2.2 and Veo 3.1.
  • Gen-ViRe Benchmark: CollabVR boosted the performance of the open-source VBVR-Wan2.2 from a baseline score of 0.391 to 0.531, and the closed-source Veo 3.1 from 0.481 to 0.550.
  • VBVR-Bench: CollabVR improved VBVR-Wan2.2's overall score from 0.671 to 0.757, achieving the highest gains in categories requiring multi-step spatial or transformation reasoning.
  • VLM Generalization: While Gemini 2.5 Pro served as the default supervisor, smaller open-source models like Qwen3.5-9B paired with CollabVR still outperformed proprietary models running standard Pass@$k$ baselines.

Limitations and Future Work

Despite its success, CollabVR is bounded by the underlying capabilities of the VGM. It cannot "manufacture" capabilities the generator completely lacks; for instance, symbolic or abstract transformations (like Raven's Progressive Matrices) see smaller gains because the VGM struggles to approximate them even with perfect prompts. Additionally, the VLM verifier is not infallible and can occasionally allow failed clips to pass silently.

The authors suggest that future work should focus on integrating reasoning-oriented VGM training (such as physics-aware fine-tuning) and developing finer-grained failure localization techniques within the test-time loop.

Summary updated

Abstract

Recent "Thinking with Video" approaches use Video Generation Models (VGMs) for visual reasoning by producing temporally coherent Chain-of-Frames as reasoning artifacts. Even strong VGMs, however, exhibit two recurring failure modes on goal-directed tasks: long-horizon drift on multi-step tasks and mid-clip simulation errors that compound. Both stem from the absence of explicit reasoning built upon the VGM's short-horizon visual prior, a role naturally filled by Vision-Language Models (VLMs), but where to place the VLM is non-trivial: upfront plans commit before any frame is generated and post-hoc critiques over whole videos intervene too late. We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level granularity: the VLM plans the immediate next action, inspects the clip the VGM generates, and folds the verifier's diagnosis directly into the next action prompt to repair detected failures. On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute, with the largest gains on the hardest tasks. It also yields further improvements on top of a reasoning-fine-tuned VGM, indicating that step-level VLM supervision is orthogonal to and stackable with reasoning-oriented fine-tuning. We provide video samples and additional qualitative results at our project page: https://joow0n-kim.github.io/collabvr-project-page.
BibTeX
@misc{kim2026collabvr,
  author = {Joowon Kim and Seungho Shin and Joonhyung Park and Eunho Yang},
  title = {{CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models}},
  year = {2026},
  eprint = {2605.08735},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.08735},
}

Loading related content...

Figure