Finding the few seconds of an hour-long video that answer a question is usually done coarse-to-fine. A cheap pass skims the video and proposes candidate time windows, then an expensive vision-language model (a model that reads frames and text together) zooms in on those windows to answer. The weak point is the handoff. If the skim proposes the wrong window, or the zoomed model hallucinates that the queried event happens there, nothing downstream catches it. The usual fix is to add a judge, meaning another language model that scores the candidate. But multimodal judges are known to invent visual details while reasoning and then defend their verdicts with those inventions.
We propose to make the check adversarial and tie it to evidence. When coarse proposals disagree, agents argue over which window, and which boundaries, are correct. There is one rule: an agent can only argue by requesting a probe. A probe is a short time window plus a yes/no visual question, and a separate perception model with no access to the argument answers it on fresh high-resolution frames. The judge never reads the agents' prose. It reads only the list of probes and their answers. In two sentences: debate decides where to spend expensive zoom frames, and a skeptic is rewarded for finding frames that contradict the leading proposal. Because the judge sees only answers to probes, the persuasive-but-unfounded arguments that plague debate cannot sway it.
Core Claim
Under a fixed budget of high-resolution frames, adversarially chosen probes find more disconfirming evidence than confidence-driven zooming or independent re-sampling. They should therefore cut false-positive localizations, where the model reports an event in a clip that does not contain it, and tighten boundaries. Invoking this debate only on queries where cheap proposals disagree should preserve accuracy on easy queries.
The claim is falsifiable. If a single verifier given the same probe budget, choosing its own probes to reduce its own uncertainty, matches the debate, then adversarial probe selection adds nothing. In that case the useful part is only the evidence-bound judge, and we will report exactly that.
Why This Is Different From What Exists
Several systems already combine localization with task decomposition, and they motivate the gap. MUPA runs grounding-first, answer-first, and joint reasoning paths in parallel, then merges their timestamps with a reflective agent and weighted k-means. LongVideoAgent trains a master agent to call grounding and vision agents cooperatively. MAGNET sends independent audio-visual agents' findings forward to a GPT-4o meta-agent. FALCONEye lets an LLM trigger recursive clip zooms based on a VLM's token-probability confidence. In all four, information flows one way into an aggregator, and no component is tasked with trying to break the leading hypothesis. Our system does not aggregate independent paths after the fact. Instead, it makes one agent's job the search for contradicting frames.
Debate supplies the mechanism but also constrains the design. Formal refutation rounds (FORD) and skeptic-versus-trusting agents for hallucination detection (Sun et al.) show that assigned adversarial roles help. A Diagnostic Study of Multi-Agent LLMs for Real-World Debates shows that judges of free-text debates miss groupthink and persuasive unfounded claims, and Adversarial Review finds diminishing returns from unstructured agent scaling. We take both negative results seriously. Video localization is a domain where claims can be checked against frames, so we force every debate move to be a checkable probe.
The judge design responds directly to MLLM-as-a-Judge. That paper reports that chain-of-thought degrades multimodal judging because judges hallucinate perceptual premises. It also reports that judges do better with textual descriptions than with raw visual inputs, and that pair comparison is the most reliable format, at about 79% human agreement for GPT-4V. Our judge therefore compares two candidate spans pairwise, using only short textual probe answers, and never perceives the video itself.
Two further lines of work supply parts of the method. Step-level judging of intermediate nodes, which AgentEval validated offline (failure-detection recall 0.89 versus 0.41 end-to-end), becomes an online gate on coarse proposals. The lesson from MAgICoRe is that refining everything hurts: it reports a 5.2% drop on MATH under universal refinement. That lesson becomes our routing rule.
Research Program
We will build a training-free pipeline around a single open backbone, Qwen2.5-VL-7B, so that every comparison uses the same model.
Coarse stage. A low-frame-rate skim, in the style of Video-in-the-Loop, is combined with HiTeA-style candidate pruning (HiTeA). Together they produce $K$ candidate spans.
Routing. If the top candidates overlap strongly and the cheap scorer's margin is high, the pipeline answers directly. Otherwise the query enters debate.
Debate. Proponent agents each defend one candidate span. A skeptic, rotated across candidates, attacks the current leader. Each turn is exactly one probe, for example "between 412.0s and 414.0s, is the person already holding the knife?" A perception instance runs the probe on dense frames from that window only, and it sees neither the debate nor the original question's answer options.
Boundary disputes become probes near the claimed start and end times. This naturally produces an adversarial bisection: the skeptic probes just inside a claimed boundary to show the event has not started, and the proponent probes to show it has. The debate stops when the frame budget $B$ is spent or the judge's pairwise preference stabilizes across two rounds.
Judge. The judge receives the probe log, meaning windows, questions, and answers, and outputs a pairwise preference between spans plus refined boundaries. It may also output "not present." This echoes the explicit absence supervision in ReVisionLLM.
Analysis. Beyond accuracy, we will measure three things. The first is how often skeptic probes return disconfirming answers, compared with the probes a confidence-driven zoomer chooses. The second is where false positives survive. The third is the wall-clock latency of the added rounds, a known cost of multi-pass pipelines (A.I.R.).
Baselines
Naive reference. This is single-pass uniform sampling with the same backbone at the total frame count our full system consumes. It shows what spending the frames without any search buys. Beating it means higher localization and answer accuracy at an equal frame count.
Strongest existing method on grounded question answering. This is MUPA. It is the closest multi-agent system that jointly answers and grounds, and it reports Acc@GQA of 30.3% on NExT-GQA with MUPA-7B and 47.4% on DeVE-QA. We will compare against the reported numbers. Where code allows, we will also rerun it with our backbone and frame budget, because MUPA's fine-tuned LoRA agents differ from our training-free setup. Beating it means exceeding 30.3% Acc@GQA on NExT-GQA at a comparable model size.
Strongest existing method on pure temporal grounding. This is Video-in-the-Loop. It reports zero-shot mean IoU on Charades-STA rising from 42.3% to 54.0%, and 47.4% accuracy on LVBench. On Ego4D-NLQ the reference is HiTeA's training-free mIoU of 8.12%. On MAD, ReVisionLLM reports only a relative +2.6% gain in [email protected], and we found no absolute figure on Lacuna for it. We will run our method on the same splits and metrics.
Compute-matched single verifier. This is the ablation that isolates our key ingredient. One agent receives the same routing, the same probe interface, the same budget $B$, and the same probe-only judge. It chooses probes to resolve its own uncertainty, in the manner of FALCONEye's or A.I.R.'s confidence-driven zoom, rather than to defeat a rival. The only thing removed is the adversary.
Free-text debate. A second ablation keeps the adversary but lets the judge read the agents' arguments and allows claims without probes. This isolates evidence binding.
Universal debate. A third ablation removes routing and sends every query to debate. This tests whether MAgICoRe's excessive-refinement effect transfers to video.
For all baselines, tuning budgets are equal: prompt search is limited to the same number of development-set trials per system.
Why not just…?
Why not just have one strong verifier check each candidate? This is the main threat. A careful verifier with a dense zoom may catch most false positives, and prior work shows that re-looking at frames at higher fidelity drives the gains: Video-in-the-Loop's "think twice" ablation found reasoning alone helped little, while token reallocation helped a lot. Our compute-matched single-verifier ablation settles this directly. If it matches the debate on false-positive rate and IoU, we drop the adversary and report that the evidence-bound judge alone is the contribution. We expect the gap to appear specifically on hard negatives, meaning visually similar but wrong windows. There a verifier's self-chosen probes tend to confirm its prior, which is the self-confirmation weakness noted for VideoAgent2.
Why not just sample many proposals and vote or cluster? Self-consistency is cheap and parallel, and MUPA's weighted clustering is a strong version of it. However, MAgICoRe reports that voting hits an "aggregation rut": one refinement iteration beat 120-way self-consistency by 3.4% at about half the compute. In video, independent samples share the same coarse-skim blind spots, so they tend to agree on the same wrong window. We will include a baseline that clusters 16 or more proposals at an equal frame budget, and we will report results on the subset where the majority is wrong.
Why not just feed the whole video to a long-context model? Reported numbers argue against this at our scale. On LVBench, Video-in-the-Loop's 47.4% exceeds GPT-4o's 27.0% and Gemini-1.5-Pro's 33.1%. On LongTVQA, LongVideoAgent's 82.65% exceeds GPT-4o's 70.78%. FALCONEye reports a cost of $0.01 per query versus $0.11 for GPT-4o. We will still include Gemini-2.5-Pro on a sampled subset as a cost-accuracy reference point rather than as a primary competitor.
Proposed Experiments
The main evaluation covers Charades-STA and Ego4D-NLQ, measured by mIoU and Recall@IoU; NExT-GQA, measured by Acc@GQA and Intersection-over-Prediction (the fraction of the predicted span that falls inside the true evidence span); MAD for hour-scale grounding; and FALCON-Bench, where localization and open-ended answering are joint.
We add one constructed test that none of these benchmarks covers. For each Charades-STA and Ego4D-NLQ query, we pair it with a video that does not contain the queried event, using hard negatives from the same activity class. We then measure how often each system correctly outputs "not present." This directly targets the calibration failures reported by ReVisionLLM and FALCONEye.
The decisive checks are three. First, at equal $B$, does debate beat the single verifier on both false-positive rate and boundary IoU? Second, do skeptic probes return disconfirming answers significantly more often than verifier probes? That is the proposed mechanism, and if it fails, the accuracy result has no explanation. Third, does routed debate beat universal debate on easy queries while matching it on hard ones?
We will report every accuracy number together with frame count and wall-clock latency. A gain that disappears once compute is matched counts as a negative result.
References
- MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering
- LongVideoAgent: Multi-Agent Reasoning with Long Videos
- MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
- FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMs
- Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
- Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
- A Diagnostic Study of Multi-Agent LLMs for Real-World Debates
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
- Video-in-the-Loop: Span-Grounded Long Video QA with Interleaved Reasoning
- HiTeA: Hierarchical Temporal Alignment for Training-Free Long-Video Temporal Grounding
- ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos
- A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering
- VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT