Cross-Level Consistency as a Free Label Check for Synthetic Video Grounding Data

Spatio-temporal video grounding means finding, from a sentence such as "the man in the grey coat picks up the bag and walks to the door," both when the described thing happens (a start and end time) and where the target is in every frame of that interval (a sequence of boxes called a "tube"). The best models for this need human annotators to draw boxes frame by frame. The cost grows with every frame of video, which is why benchmarks stay small: one recent multi-object benchmark required people to inspect 10.2 million frames. Automatic data pipelines scale easily, but they produce text that is often unrelated to what is on screen. In one large automatic video question-answering corpus, 39% of generated items described things that were not visible.

We propose to generate grounding data the opposite way from most synthesis pipelines: track first, describe second, then keep a description only if it agrees with itself across levels of detail. An off-the-shelf tracker extracts object tubes from real, unlabeled video. A multimodal language model describes each tube at three levels: what the entity looks like in a single frame, what it does over a time segment, and the event the whole clip shows. We keep a synthetic query only if each level, grounded independently by a frozen model, points back to the same tube and the same time span. The central bet is that agreement across levels detects mislabeled synthetic data and costs no human labels. If it works, multi-level alignment stops being only a model architecture and becomes the quality filter that makes synthetic dense video supervision usable.

Core Claim

Training on tracker-derived tubes paired with cross-level-verified hierarchical queries will close a substantial part of the gap between weakly supervised and fully supervised grounding. This should hold on VidSTG and HC-STVG with zero human box annotations. The gain should come from the consistency filter, not from data volume. A matched-size, unfiltered version of the same corpus should do markedly worse, and a human audit should show the filter cuts the visually ungrounded rate well below the 39% reported for transcript-derived data in Just Ask.

Where This Sits

Three bodies of work motivate the proposal.

The first is weakly supervised grounding, which avoids per-frame boxes but caps accuracy. WINNER parses sentences into hierarchical trees and aligns them to detector-proposed tubes using only video-sentence pairs. Video-Text Prompting draws markers around candidate tubes, scores them with a frozen CLIP, and erases candidates to measure how much the score drops. Both learn from captions written for the whole video, so they never see a sentence written specifically about one tube.

The second is synthetic multimodal data for video. Generalized Video Moment Retrieval turns language-model-extracted scene graphs into temporal queries, but it produces no boxes. SVLTA renders VirtualHome videos with exact event boundaries, but it is an evaluation benchmark with a visual style far from real footage, and it is temporal only.

The third is dense grounding systems trained on manual tubes. Video-GroundingDINO adds temporal adapters to an image grounding model. Context-Guided STVG mines instance context to refine decoding. OmniSTVG grounds every mentioned object at once. All three depend on fully annotated datasets.

Two papers supply the method rather than the motivation. STVG-R1 shows that a detector plus SAM2 tracks can stand in for coordinate annotation, as candidate tubes that a model picks from by ID. However, it still trains on labeled HC-STVG queries and does not synthesize any. GLIP shows that teacher-generated pseudo-boxes over 24M web image-text pairs can bootstrap grounding in static images. We borrow both ideas: tracker tubes as the spatial ground truth, and pseudo-labeling at scale. We differ from both by generating the language from the tube and verifying it across levels of detail, rather than generating boxes from given text.

One prior result constrains the design. Weakly-Supervised Temporal Article Grounding argues that if a detail is localized to a segment, its parent summary must also associate with that segment. We take this nesting requirement as the formal basis of the filter.

The closest existing work is STVG-R1 combined with GVMR: tracker tubes on one side, LLM-generated queries on the other. Nobody in the file joins them into tube-grounded hierarchical queries, and nobody uses hierarchy as a noise filter.

Research Program

Source video and tubes. The pipeline starts from unlabeled real video: HowTo100M clips (as used in Just Ask) and YFCC100M clips (as used by NExT-VMR). We remove any video overlapping VidOR, which underlies VidSTG, or the movie sources of HC-STVG. A detector plus SAM2 produces object tubes, and we keep tubes that persist long enough and do not switch identity, measured by appearance-embedding drift.

Hierarchical descriptions. For each video, a multimodal language model sees all tubes, each marked with an ID, and writes three descriptions for a target tube. The entity description covers appearance in one keyframe. The action description covers what the entity does, with a proposed start and end time. The event sentence is a full query combining both, written to tell this tube apart from the other tubes in the clip. That last requirement matters because HC-STVG videos mostly contain more than three people. Multi-entity queries are produced in the same way for BOSTVG-style evaluation.

The cross-level filter. Let $\tau$ be the target tube and $[t_s, t_e]$ its proposed span. A frozen scorer, CLIP with visual markers as in Video-Text Prompting, ranks all tubes in the video separately for the entity description, the action description, and the full query. A frozen temporal model localizes the action description in time. We keep a query only if all three spatial rankings place $\tau$ first by a margin $\delta$ over the runner-up. The temporal localization must also overlap $[t_s, t_e]$ with IoU above $\gamma$. Finally, erasing $\tau$ must drop the full-query score, which tests visual dependence in the spirit of contrastive erasure. The thresholds $\delta$ and $\gamma$ trade corpus size against purity, and we sweep them.

What is analyzed. We measure the precision of the filter against a human audit. We also check what the filter discards and whether it biases the corpus toward large, easily tracked objects. Because the pipeline inherits SAM2's misses, objects the tracker drops never become training targets. We will report results binned by object size and occlusion to show this cost honestly.

Grounding model. To isolate the effect of data, we hold the architecture fixed at Video-GroundingDINO and change only the training corpus.

Baselines

Naive reference points. The first is the frozen marker-prompted CLIP scorer applied directly at test time, with no training. This tests whether the filter's own scorer already solves the task, in which case the trained model would add nothing. The second is a model trained only on transcript-aligned clips, following the Just Ask recipe of using narration as weak text, with tracker tubes as candidates.

Strongest weakly supervised method. This is Video-Text Prompting with its contrastive variant, which reports 17.9 mean vIoU on VidSTG declarative queries and 16.43 mean vIoU on HC-STVG. It is the closest competitor because it also uses zero box labels and the same frozen-CLIP marker trick our filter uses. We will retrain it on our unlabeled video pool, using whole-video captions generated by the same language model, with matched GPU-hours and tuning trials. Beating it means a clearly higher m_vIoU on both benchmarks under the same label budget. We would want at least several points, given that its jump over WINNER's 11.61 was about six.

Fully supervised ceiling. The ceiling is Video-GroundingDINO. It reports 34.67 m_vIoU on closed-set VidSTG declarative queries and 39.88 on HC-STVG v2. Context-Guided STVG reports 38.4 m_vIoU on HC-STVG v1. These numbers mark how much of the human-label gap we close. They are not targets for the zero-label setting.

GLIP-style self-training. A teacher, CVTP, pseudo-labels whole-video captions on the same video pool, and the student trains on its predictions with the same compute.

Key ablations. The first is the identical pipeline with the cross-level filter replaced by a random subset of equal size. The second replaces it with a single-level filter that checks only the full-query ranking. Together these isolate whether the hierarchical agreement matters, rather than any filtering at all.

Why not just…?

Why not just render synthetic videos with perfect labels, as in SVLTA? Simulator footage has simple textures, lighting, and motion, and the file reports out-of-distribution drops when moving to real video. Diffusion video generators also change object identity mid-clip, which would corrupt tube labels. Our pipeline keeps real pixels and puts the uncertainty in the language, where it can be checked. As a side experiment, we will train on a VirtualHome-rendered corpus of matched size and compare transfer.

Why not just pseudo-label with an existing teacher, GLIP-style? This is a real threat, since GLIP's self-training worked well for images. The difference is that a teacher trained on global captions inherits their vagueness: it is never asked to tell apart three similar people. Our queries are written per tube against distractors. The self-training baseline above settles the question directly. If it matches our corpus at equal compute, the generation step is unnecessary and we would report that.

Why not just run a tracker-ID multimodal model zero-shot, as in STVG-R1? STVG-R1's reported gain is relative: 20.9% mIoU over Qwen2.5-VL-7B on HC-STVG v2. It was obtained with labeled HC-STVG training, so it says nothing about the zero-label regime, and the file contains no zero-shot absolute numbers for it on VidSTG. We will evaluate the untrained ID-marker model as a zero-label reference. Our synthetic corpus could also serve as RL training data for STVG-R1, which would be a positive outcome rather than a competing one.

Why not just filter by single-query confidence? That filter is cheaper, and it may catch most of the noise. The single-level ablation tests exactly this, and if it ties, the hierarchical claim fails.

Proposed Experiments

Zero-human-label grounding. We train on synthetic data only and evaluate on VidSTG (declarative and interrogative), HC-STVG v1 and v2, and BOSTVG. We report m_vIoU, [email protected], [email protected], and m_tIoU against the baselines above. BOSTVG's OmniTube baseline reports 9.47 m_vIoU with full supervision, so even matching it without labels would be informative.

Label efficiency. We pretrain on synthetic data, then fine-tune on 10%, 25%, and 100% of VidSTG or HC-STVG annotations. We compare against Video-GroundingDINO trained on the same fractions from scratch. The decisive check is whether synthetic pretraining plus 25% of labels matches 100% of labels.

Filter audit. Annotators judge 500 sampled queries before and after filtering for visual groundedness and tube correctness. We compare the ungrounded rate to Just Ask's 39%, and we plot downstream m_vIoU against the filter's measured precision as $\delta$ and $\gamma$ vary.

Failure accounting. We break results down by object size, occlusion, and query type. Interrogative queries are reported separately because Video-Text Prompting is known to be weaker on them. This shows where reliance on the tracker caps performance.

The proposal is falsified if the random-subset or single-level ablation matches the cross-level filter at equal corpus size. It is also falsified if the trained model fails to beat CVTP's 17.9 and 16.43 m_vIoU under matched compute.