Multi-Agent Self-Improving Reinforcement Learning for Video Reasoning

Mingwen Zhang, Jisheng Dang, Minqiang Yang, Bimei Wang, Bin Hu, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

The paper introduces MASIRL (Multi-Agent Self-Improving Reinforcement Learning), a framework that moves video verification from a post-hoc inference filter directly into the training loop. By coupling a trainable Grounder with a frozen Verifier via group-relative reinforcement learning and boundary calibration, the model achieves consistent zero-shot transfer gains across multiple video question answering and temporal grounding benchmarks.

What Problem Does This Paper Address?

When AI models reason over videos—such as answering questions grounded in specific video clips—they must not only generate the correct textual answer but also precisely localize the supporting temporal evidence.

Traditionally, models are trained using local objectives like boundary regression or span generation. If a verifier or reranking module is used, it is typically tacked on at inference time to sort through candidate segments. This creates a training-testing mismatch: during training, the model never learns how to rank multiple semantically plausible intervals against each other, often leading it to select segments that are too loose, over-inclusive, or weakly justified.

Comparison of traditional separate training vs. MASIRL's self-improving integration.

Methods

MASIRL addresses this gap using a multi-agent, self-improving reinforcement learning loop consisting of two primary actors:

  1. The Grounder ($\pi_\theta$): A trainable generative multimodal model that proposes candidate evidence segments and answers queries.
  2. The Verifier ($V_\phi$): A frozen model trained beforehand to score the relevance of video segments relative to a text query.

During training, the interaction is governed by three core mechanisms:

  • Verifier-guided Group-Relative REINFORCE (GRR): The Grounder samples a group of $G$ trajectories for a single input. The Verifier scores the resulting segments. Rather than using an absolute reward, MASIRL computes a leave-one-out within-group baseline to derive relative advantages, optimizing the language backbone via token-level negative log-likelihood (NLL).
  • Bootstrapped Boundary Calibration (BBC): To refine the Grounder's temporal head, candidate spans across the sampled group are pooled. The top-performing spans (according to the Verifier) serve as pseudo-targets to train the temporal boundary head via a localized boundary loss.
  • Verifier-Based Update Gating: To prevent training from being dominated by easy examples, updates are gated. If a sampled group already contains a trajectory exceeding a rolling validation-derived verifier threshold, its policy gradient update is zeroed out, forcing the model to focus on harder instances.
Fig. 2. Overview of the Grounder–Verifier training framework used in this paper. The Grounder generates candidate temporal segments from video and query inputs. GRR samples multiple trajectories and updates the policy from within-group relative advantages. Bootstrapped boundary calibration uses Verifier-selected pseudo-targets to refine temporal predictions. Gradients propagate only through the Grounder, while the Verifier remains frozen and decoupled from optimization.

Datasets

The model is evaluated using a strict zero-shot transfer protocol, meaning it is never fine-tuned on the target evaluation datasets.

  • Source/Pretraining Datasets:
  • Grounder: Trained on 210K samples from QVHighlights, DiDeMo, TACoS, QuerYD, HiRESTmr, HiRESTstep, CosMo-Cap, and InternVid-VTime.
  • Verifier: Trained on 232K verification triplets constructed from official training splits of DiDeMo, TACoS, and QVHighlights.
  • Target/Evaluation Benchmarks:
  • Temporal Grounding: Charades-STA, ActivityNet-Captions.
  • Grounded Video QA: NExT-GQA.
  • Long-Video/General Video QA: Video-MME, MLVU, LVBench.

Key Insights and Ideas

  • Verification as a Training Signal: The core "aha" moment is that a frozen verifier can provide a stable, sequence-level preference signal during training without the instability or reward hacking often seen when jointly updating a critic model.
  • Relevance vs. Precision Trade-off: Because the Verifier is optimized to judge semantic relevance rather than exact frame boundaries, the reinforcement learning signal preferentially boosts metrics like mean Intersection-over-Union (mIoU) and loose recall (e.g., [email protected]) rather than strict boundary metrics ([email protected]).
  • The Power of Group Comparison: Adapting group-relative policy optimization (similar to GRPO) to multimodal grounding allows the model to effectively stabilize sequence-level updates without needing a complex, learned reward critic.

Key Results

Evaluated at a 2-billion parameter scale, MASIRL delivers modest but highly consistent improvements over its direct baseline, VideoMind, across all zero-shot transfer tasks:

  • NExT-GQA: Achieved 28.7 mIoU and 25.4% Acc@GQA (an improvement of +0.1 mIoU and +0.2% accuracy over the baseline).
  • Temporal Grounding: Reached 46.1 mIoU on Charades-STA and 30.5 mIoU on ActivityNet-Captions.
  • Long-Video QA: On Video-MME, it achieved 54.1% overall accuracy and 45.9% on long videos specifically, outperforming same-scale baselines like VideoMind and LongVA.
Fig. 3. General Video QA comparison and gain margin. Multiple baselines are shown for Video-MME, MLVU [39], and LVBench [40]; in each panel, methods scoring above MASIRL on that benchmark are omitted so that the margins against non-superior baselines remain readable. The MASIRL bars are highlighted, while the legend and value labels are placed outside the plotted areas to avoid occlusion.

Limitations and Future Work

  • Boundary Precision Weakness: The current framework struggles with fine-grained boundary adjustment. It frequently suffers from "early-start" or "late-end" errors because the Verifier does not penalize extra context frames as long as the core action is present.
  • Computational Overhead: Generating multiple rollouts per video query and repeatedly running them through a secondary Verifier model makes the training loop significantly more expensive than standard supervised fine-tuning.
  • Modest Margins: While directionally consistent across all benchmarks, the absolute performance gains over a strong baseline remain relatively narrow. Future work will explore incorporating boundary-sensitive negative samples during verifier training to bridge the gap between semantic relevance and exact temporal precision.

Summary updated

Abstract

Video reasoning tasks such as grounded video question answering and temporal grounding require selecting temporal evidence that supports the query. In many current training setups, temporal supervision is applied through local objectives such as boundary regression or span generation, while verification is used mainly to rerank candidate segments at inference time. We study whether a frozen verifier can also guide training. Our multi-agent framework couples a trainable \emph{Grounder} with a frozen \emph{Verifier}: the Grounder samples candidate trajectories and evidence segments, the Verifier assigns query-conditioned segment scores, a group-relative policy-gradient objective favors trajectories that outperform their within-input peers, and a bootstrapped calibration loss steers temporal predictions toward verifier-preferred spans. Trained on source tasks and evaluated without target-dataset fine-tuning, a two-billion-parameter instantiation transfers zero-shot across grounded question answering, temporal grounding, and long-video question answering, reaching 28.7\% intersection-over-union and 25.4\% answer-grounding accuracy on a grounded-question-answering benchmark, 46.1\% intersection-over-union on a temporal-grounding benchmark, and 54.1\% on a long-video question-answering benchmark. Relative to a strong same-scale baseline, the gains are modest but consistent, with the clearest improvements on relevance-oriented metrics such as intersection-over-union and moderate-overlap recall. Within the tested benchmarks and transfer setting, the results support frozen verification as a training signal for evidence selection, while showing that strict boundary precision remains comparatively weaker. Code and models are available at https://anonymous.4open.science/r/MASIRL-E50C/
BibTeX
@misc{zhang2026multi,
  author = {Mingwen Zhang and Jisheng Dang and Minqiang Yang and Bimei Wang and Bin Hu and Tat-Seng Chua},
  title = {{Multi-Agent Self-Improving Reinforcement Learning for Video Reasoning}},
  year = {2026},
  eprint = {2608.28675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.28675},
}
Figure