Picking the Right Video by Checking Its Timeline

Video generators can be asked for "the red ball rolls left, stops, and only then does the blue cube start sliding toward it." They usually return something that looks right in any single frame but gets the order of events wrong, the durations wrong, or both. We propose a simple test-time fix that can also be measured honestly. First, sample $N$ videos from the generator. Then track the objects in each video, turn the tracks into trajectories, and score each trajectory against a formal timeline specification written in signal temporal logic, a language for statements like "A holds for at least two seconds before B begins." The video that best satisfies the specification is returned. This is best-of-$N$ sampling (generate many candidates, keep the one a scorer likes most), but the scorer checks the timeline rather than appearance or a language model's opinion.

The second half of the proposal is the evaluation. Static video benchmarks can be memorized and gamed, and frontier video models answer many of their questions without looking at the frames at all. We therefore generate test specifications procedurally and freshly at evaluation time, in the style of DyVal's random dependency graphs. Each specification is compiled into both a text prompt and a ground-truth logic formula. The result is a contamination-resistant benchmark on which the order of events provably matters. The two halves together let us answer two questions separately. Does the generator ever produce a correctly timed video (its ceiling)? And can a given scorer find that video (verifier quality)?

Core Claim

For prompts whose correctness depends on the order and duration of events, the bottleneck in best-of-$N$ video generation is the verifier, not the generator.

We make two specific predictions. First, general vision-language judges will give almost no selection gain on order-dependent prompts. The reason is that they score shuffled and unshuffled frames nearly the same, which Breaking Down Video LLM Benchmarks observed for GPT-4o and Gemini-1.5-Pro on video question answering. Second, a trajectory-plus-temporal-logic verifier will recover a large share of the gap between single-sample accuracy and the generator's pass@$N$ ceiling.

Best-of-$N$ also stays close to the base model's distribution. The classical bound on its divergence from the reference policy grows only like $\log N$, and Best-of-N through the Smoothing Lens analyzes this regime. So we expect correct timing without the flicker and visual collapse that TTOM reports when attention maps are forced directly.

Positioning Against the Closest Work

Four lines of work come closest, and each leaves a gap this proposal fills.

Test-time steering of the generator. TTOM optimizes lightweight adapter weights during early denoising so that attention maps match an LLM-drawn bounding-box layout. MotiMotion has a vision-language model plan trajectories, including secondary causal effects, before a single generation pass. Neither samples a pool and rejects candidates. Neither checks event order or durations after generation, since TTOM's objective is spatial alignment per frame.

Best-of-$N$ with a multimodal verifier. DiscPRM supplies the method template: a learned verifier for best-of-$N$ with multimodal inputs. It is confined to static images and discrete image operations such as crops and sketches.

Formal temporal logic as a target. Diverse Controllable Diffusion Policy With Signal Temporal Logic and Reward Machines for Signal Temporal Logic show that such specifications can drive trajectory generation in control. They have not been used as rerankers for pixel-space video.

Dynamic benchmarks. DyVal and A²RBench provide procedural, verifiable generation and symbol remapping, but only for text and grids.

The new element is coupling three things:

  1. a temporal-logic verifier over tracked trajectories,
  2. best-of-$N$ selection,
  3. a procedurally generated benchmark whose ground truth is the same kind of formula.

We also explicitly keep the selector separate from the final judge.

Research Program

The benchmark (ChronoSpec)

A generator samples a small directed graph of events. Each event has:

  • an object identity,
  • a motion primitive (enter, translate in a direction, stop, rotate, contact),
  • precedence edges (B starts after A ends),
  • duration bounds (A persists at least $\tau$ seconds).

Graph depth and width control difficulty, following DyVal's complexity tiers. Each graph is rendered two ways: as a text prompt through templated and LLM-paraphrased wording, and as a signal temporal logic formula $\varphi$ over object-centroid and velocity signals.

To expose shortcuts, we use three variants:

  • Remapped variants swap object nouns and colors, borrowing A²RBench's remapping test.
  • Reversed-description variants list events out of chronological order in the prompt, borrowing DyVal's order manipulation.
  • Order-flip pairs are prompts that share every object and motion but differ only in event order.

On order-flip pairs, any scorer that ignores time must score both videos identically. This is our built-in shuffle-invariance probe. Fresh instances are drawn for every evaluation run, so nothing about the test set can be memorized.

The verifier

Each candidate video is processed by an open-vocabulary detector and point tracker of the kind T2V-CompBench already uses (GroundingDINO and DOT). This produces per-object trajectories $x(t)$. We then compute the quantitative robustness $\rho(\varphi, x) \in \mathbb{R}$ of formula $\varphi$ on trajectory $x$. Robustness is positive exactly when $x$ satisfies $\varphi$, and its magnitude measures the margin. Selection is

$$\hat{v} = \arg\max_{i \le N} \; \rho(\varphi, x_i) + \lambda\, q_i,$$

where $x_i$ is the trajectory extracted from candidate video $v_i$, $q_i$ is a per-frame visual quality score, and $\lambda$ is tuned on a held-out split.

For natural-language prompts that have no ground-truth formula, an LLM compiles the prompt into $\varphi$. This compile step is a known weak point: ChatGPT reached only 52.3 F1 on event-event ordering in TEMPREASON. We therefore report results with oracle formulas and with compiled formulas separately, so the loss from compilation is visible.

As a compute saver, we apply the verifier to the denoiser's predicted clean video at an intermediate step and drop low-robustness candidates early, in the spirit of Speculative Rejection.

Generators

We use Wan2.1 and CogVideoX, the two generators on which TTOM reports results. We use both text-only conditioning and trajectory-conditioned variants, the latter where Time-to-Move's training-free dual-clock denoising can be applied.

Baselines

Naive reference: single sample. One sample from each generator at default settings ($N=1$). It anchors every gain.

Generator ceiling: oracle pass@$N$. This is the fraction of prompts for which at least one of $N$ candidates satisfies $\varphi$ according to the held-out judge. It is not a method but an upper bound. Every selector is reported as the fraction of the gap between single-sample accuracy and pass@$N$ that it closes.

Strongest existing method: TTOM. TTOM uses test-time adapter optimization against LLM layouts with its motion memory, run on Wan2.1 and CogVideoX. On T2V-CompBench it reports a 14% relative improvement on Wan2.1 and 34% on CogVideoX, and an 82.57% improvement in the motion category on Wan2.1 (TTOM). We will rerun it with the authors' settings rather than copying numbers. Our method is given exactly TTOM's measured wall-clock GPU budget per prompt, which fixes $N$. Beating TTOM means higher formula-satisfaction rate on ChronoSpec order-flip pairs at matched compute, with no loss on VBench quality, and parity or better on T2V-CompBench motion and dynamic-attribute categories.

Strongest selection competitor: vision-language judge. Best-of-$N$ where a strong vision-language model (Qwen2.5-VL-7B, the backbone of DiscPRM, plus GPT-4o) rates each candidate's adherence to the prompt. It uses the same pool, the same $N$, and an equal prompt-tuning budget on the held-out split. No reported result found on Lacuna for vision-language-judged best-of-$N$ on video generation. The nearest number is DiscPRM's 65.9 F1 on the static-image VABench versus 59.9 for VisualPRM (DiscPRM).

Planning baseline. A MotiMotion-style setup: an LLM-expanded prompt plus a planned trajectory and a single sample. MotiMotion reports a 78.0% overall 2AFC preference win rate on MotiBench (MotiMotion). We apply our selector on top of planning as well.

Ablations isolating the key ingredient. We run four:

  1. the same tracker, but scored with an order-agnostic set-membership check (are all required motions present anywhere) instead of the temporal formula;
  2. a verifier with oracle formulas versus one with LLM-compiled formulas;
  3. tracker swaps;
  4. the effect of early rejection.

The first ablation tests whether ordering and duration checks, rather than tracking itself, produce the gain.

Separating the selector from the judge. The selector and the judge are never identical. Final scores come from a differently configured tracking pipeline and from human annotation of event order on a sampled subset. Gains that appear only under the selector's own tracker count as reward hacking, not progress.

Why not just…?

Why not just use a strong vision-language model as the best-of-$N$ judge? This is the real threat, because it needs no specification and handles open-ended prompts. The evidence against it is indirect but pointed: frontier video models barely change their answers when frames are shuffled (Breaking Down Video LLM Benchmarks), which suggests they cannot rank videos that differ only in order. ChronoSpec's order-flip pairs settle the question directly. If the vision-language judge closes a comparable share of the pass@$N$ gap on order-flip pairs, our verifier's advantage is confined to precision on durations, and we would say so.

Why not just use T2V-CompBench's existing tracker metrics as the selector? Those metrics already track objects and check motion direction, so they are a cheap, strong reranker. However, they score per-object motion alignment rather than cross-event precedence and persistence. Our first ablation, which keeps the tracker but removes temporal operators, is exactly this baseline. If it matches the full verifier, the contribution reduces to the benchmark, and we would report that.

Why not just spend the compute on test-time optimization or a bigger model? TTOM shows optimization works on compositional prompts. But it optimizes per-frame spatial layouts, and it notes attention forcing can cause flicker. Scaling alone left Wan2.1-14B failing on motion until TTOM was applied. The matched-compute comparison above decides this. We also test the hybrid, TTOM followed by our selector, because the two may be complementary rather than competing.

Why not just evaluate on T2V-CompBench and VBench with contamination filters? Fixed prompt sets are learnable. DyVal found models that perform well on static benchmarks scoring near zero on freshly generated tasks, and A²RBench found a 17.7% drop under symbol remapping. We still report T2V-CompBench and MotiBench for comparability, but conclusions about timing rest on fresh ChronoSpec draws.

Decisive Checks

Check 1: does order matter to the judges? On order-flip pairs, measure how often each selector's choice changes when the target order flips. A time-blind scorer will sit at chance. Our verifier should flip almost always on candidates that the human judges say are correctly ordered.

Check 2: verifier versus generator ceiling. Plot gap-closed against $N$ (from 1 to 32) and against graph depth. If pass@$N$ itself collapses at depth three or more, the generator, not the verifier, limits the method. That outcome would redirect the work toward trajectory conditioning with Time-to-Move rather than better selection.

Check 3: does it survive natural prompts? Repeat with LLM-compiled formulas on T2V-CompBench's motion and dynamic-attribute prompts and on MotiBench. The cost of compilation is measured as the drop from oracle-formula results.

Check 4: shortcuts. Remapped and reversed-description variants should leave our selector's satisfaction rate unchanged. A large drop would show it, too, rides on familiar wording.