DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding

Zhengbo Zhang, Mark He Huang, Zhigang Tu, Ming-Hsuan YangarXiv 2026 · arXiv

TL;DR

Zero-shot video temporal grounding methods often struggle with complex queries that require multi-stage temporal or causal reasoning. DART (Difficulty-Adaptive Routing for Temporal Grounding) bridges this "reasoning gap" by using a query-conditioned Determinantal Point Process (DPP) to simultaneously select keyframes and measure query complexity via spectral entropy. Simple queries are fast-tracked for direct prediction, while complex queries undergo a structured "Temporal Markup Prompting" pipeline, achieving state-of-the-art performance while processing over 7× fewer frames.

What Problem Does This Paper Address?

Video Temporal Grounding (VTG) involves locating the start and end timestamps of a specific event within an untrimmed video based on a natural language query. Doing this in a zero-shot setting—without task-specific training data—is highly desirable but notoriously difficult.

Current state-of-the-art zero-shot methods rely heavily on frame-query feature matching. While this works well for simple actions ("a person sits down"), it falls flat on complex, multi-stage queries ("He dismounts after flipping and lands on the mat"). The authors identify this as a systematic reasoning gap: feature-matching methods score frames independently, failing to capture cross-frame dependencies like temporal ordering and causal relationships.

Fig. 1: Reasoning gap in zero-shot VTG. Left: a qualitative example from Activ- ityNet Captions [24] whose query requires temporal ordering. Feature-matching meth- ods (TFVTG [72], TAG [27]) match only “land,” missing the earlier flipping phase, while DART localizes the full event. Right: mIoU evaluated on 100 simple and 100 complex queries sampled from the ActivityNet Captions val_2 split. Feature-matching methods exhibit a large performance gap between simple and complex queries; DART reduces this gap by nearly half through difficulty-adaptive reasoning.

Methods

DART introduces a dual-path framework that adaptively decides when to deploy heavy-duty reasoning based on the complexity of the video-query pair.

  1. Evidence Extraction & Budgeting: Instead of uniformly sampling frames, DART uses a query-conditioned Determinantal Point Process (DPP) kernel. This mathematical framework selects a compact subset of keyframes that maximizes both relevance to the text query and visual-temporal diversity. An adaptive stopping criterion automatically determines the optimal frame budget for each query.
  2. Difficulty Routing via Spectral Entropy: The core innovation lies in using the eigenspectrum of the DPP kernel to measure query difficulty. If the query-relevant event structure is simple, energy concentrates in just one or two dominant eigenvalues. If it is complex and multi-stage, energy spreads across multiple eigenvalues. DART computes the spectral entropy of this spectrum to route queries.
  3. Dual-Path Temporal Localization:
  4. Fast Path (Low Entropy): Simple queries bypass complex reasoning. The selected keyframes and timestamps are fed directly to a Large Vision-Language Model (LVLM) for immediate timestamp prediction.
  5. Slow Path (High Entropy): Complex queries trigger Temporal Markup Prompting (TMP). This structured, three-step Chain-of-Thought (CoT) procedure forces the LVLM to perform:
  6. Step 1: Global Temporal Reasoning (breaking the query into key sequential/causal actions).
  7. Step 2: Per-Frame Temporal Role Annotation (labeling every keyframe as BEFORE, START, DURING, END, or AFTER under a strict monotonic constraint).
  8. Step 3: Boundary Extraction (deriving timestamps directly from the labels).
Overview of the DART pipeline

Datasets

The authors evaluated DART on two standard benchmark datasets:

  • ActivityNet Captions: Comprising roughly 20K videos; evaluations were conducted on the standard val_2 split.
  • Charades-STA: Consisting of 12,408 training and 3,720 testing video-query pairs.

To test robustness, they evaluated DART under both Identically Distributed (IID) and multiple Out-of-Distribution (OOD) settings, including moment-shifted splits, distribution-shifted Charades-CD, and novel-text Charades-CG.

Key Insights and Ideas

  • Unifying Evidence and Difficulty: The ultimate "aha moment" of the paper is that keyframe selection and difficulty estimation can be unified under a single DPP kernel. The same relational structure used to pick the most informative frames naturally exposes how many distinct stages an event contains.
  • Spectral Entropy vs. Frame Count: Simply counting frames is a poor proxy for difficulty (e.g., a simple walk across changing rooms yields many frames but requires zero reasoning). Spectral entropy measures the effective number of dominant components, making it a highly reliable indicator of actual semantic and causal complexity.
Spectral contrast between simple and complex queries
  • Constrained Chain-of-Thought: Generic CoT ("let's think step by step") often causes LVLMs to generate verbose, unfocused responses. Forcing the model to output a strict, monotonically non-decreasing sequence of temporal roles ensures temporal consistency and drastically reduces boundary errors.

Key Results

  • State-of-the-Art Performance: DART achieves top-tier zero-shot performance, outperforming the previous best zero-shot method (TAG) by +3.24 mIoU on Charades-STA and +3.34 mIoU on ActivityNet Captions. It even beats several weakly supervised methods.
  • Bridging the Reasoning Gap: On complex queries, DART reduces the performance gap by nearly half compared to traditional feature-matching baselines.
  • Massive Efficiency Gains: Because the DPP kernel filters out redundant frames, DART operates on 7× fewer frames than typical baselines (averaging just 12 frames per video instead of 86). This allows DART with adaptive routing to clock in at an efficient 3.9 seconds per query on a single NVIDIA A100 GPU—faster than fixed-pipeline alternatives like TFVTG (4.7s).

Limitations and Future Work

The authors acknowledge that high-quality keyframe selection is a strict prerequisite for their framework; if the initial keyframes fail to capture the event due to extreme noise or visual anomalies, the downstream routing and structured reasoning provide little benefit.

Additionally, the framework relies on fixed hyperparameters for the adaptive stopping threshold ($\varepsilon$) and routing threshold ($\theta$). Future work could explore dynamically adjusting these thresholds based on video domain or length characteristics.

Summary updated

Abstract

Zero-shot video temporal grounding (VTG) localizes events in untrimmed videos from natural language queries without task-specific training. Existing methods rely on frame-query feature matching, which suffices for simple events but struggles with complex multi-stage queries that require understanding temporal ordering and causal structure -- a disparity we call the reasoning gap. We propose DART (Difficulty-Adaptive Routing for Temporal Grounding), which bridges this gap by coupling difficulty-aware routing with structured reasoning in large vision-language models. A query-conditioned Determinantal Point Process (DPP) serves a dual role: selecting diverse, query-relevant keyframes as temporal evidence, and providing spectral entropy as a difficulty indicator. Simple queries are routed to a Fast path for direct prediction, while complex queries follow a Slow path with Temporal Markup Prompting, which decomposes localization into global event analysis, per-frame temporal role annotation, and boundary extraction. On Charades-STA and ActivityNet Captions, DART achieves state-of-the-art zero-shot performance across both identically distributed and multiple out-of-distribution settings, improving mIoU by up to 3.5 points over the strongest baseline while using over 7 times fewer frames. The project homepage is available at https://dart-vtg.github.io/.
BibTeX
@misc{zhang2026dart,
  author = {Zhengbo Zhang and Mark He Huang and Zhigang Tu and Ming-Hsuan Yang},
  title = {{DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding}},
  year = {2026},
  eprint = {2607.00672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.00672},
}
Figure