Serial Reproduction Reveals the Temporal Priors of Video-Language Models

Video-language models often cannot tell "pour the batter, then flip the pancake" from the reverse. Benchmarks show this failure, but they do not show what the model believes instead. This proposal borrows a tool from cognitive science to find out. In serial reproduction, each learner reproduces what the previous learner produced. Run long enough, such a chain stops reflecting its starting point and settles on the learner's own preferences, its prior. We build such chains for video. A fixed set of real clips is reassembled at each step in whatever order and durations the model reported for the previous video. The orderings the chain drifts toward are, in a measurable sense, the model's default beliefs about how events unfold. Measuring them should explain why order mistakes happen, and it should tell us which training examples would correct them.

The core claim has two parts, and both can be checked. First, the drift target of the chain is a structured object, not uniform noise. It may be a script-like canonical order, a preference for equal durations, or a pull toward the order in which captions were listed. A video's distance from this target should predict, item by item, whether the model gets its order wrong on existing benchmarks, and it should do so better than cheaper predictors. Second, contrastive fine-tuning that uses the orderings the model drifts toward as hard negatives should beat time-reversal negatives at the same compute budget. Hard negatives here are wrong answers chosen to resemble the right one.

Why this intersection, and what is already done

Iterated learning formalizes a chain of learners as a Markov chain over hypotheses. Iterated learning: Intergenerational knowledge transmission reveals inductive biases and Connecting human and machine learning via probabilistic models of cognition show that when each learner samples from its posterior, the chain's long-run distribution equals the learners' prior. How memory biases affect information transmission: A rational analysis of serial reproduction and Evaluating models of robust word recognition with serial reproduction use the same chains as a measuring instrument for memory and speech perception. They supply our method. We move it from human participants to video models, and from static stimuli to event order. Work on regularization in transmission, The evolution of frequency distributions: Relating regularization to inductive biases through iterated learning, gives a further reason to run a chain rather than probe once. Weak per-step biases that a single probe can barely see grow over generations.

Machine-learning uses of iterated learning have gone a different way. They try to induce structure by retraining fresh students. Iterated learning for emergent systematicity in VQA cleans up the programs used by neural module networks, and Improving Compositional Generalization using Iterated Learning and Simplicial Embeddings adds a discrete bottleneck to representations. Two findings from these papers constrain our design. Repeated full retraining was too costly, so the VQA work fell back to partial resets. Iterated learning also failed on unconstrained continuous representations, and the SEM-IL authors note their method struggles with continuous factors. Our design avoids both problems. Chains run at inference time with no retraining. The chain also lives in a discrete space, the permutations of a few clips plus durations snapped to time bins, so continuity never enters.

The closest connected paper is ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning. Despite its name, it alternately trains a frame selector and a reasoner, using answer accuracy as the reward. Neither model's output becomes a fresh learner's input, and it never studies what the models prefer. The obvious "iterated learning for video" idea is retraining students on teacher-generated grounded captions. We considered it and rejected it, because a skeptic's alternative is very likely to match it: plain repeated self-distillation, without the reset and bottleneck, is cheaper and probably equivalent.

The motivation comes from diagnostic work. TempCompass: Do Video LLMs Really Understand Videos? finds near-random accuracy (25–33%) on speed and direction, against 94–99% for humans, and finds that apparent success on action questions comes from static appearance. Test of Time: Instilling Video-Language Models with a Sense of Time finds roughly 50% order consistency in foundation models. Dynamic Reflections: Probing Video Representations with Text Alignment shows that language-model embeddings barely distinguish "A before B" from "B before A." These papers show that models fail. Our chains ask what they fall back on.

Research Program

The chain. We start from untrimmed videos with $K$ annotated events: ActivityNet Captions for coarse events, Charades-STA for daily activities, and TEMPO-style closely spaced events where they are available. From each video we cut the $K$ event clips. Generation 0 assembles them in their true order and durations.

At each later generation, the model watches the current assembly and reports an ordering and a duration bin for each event. The next assembly is built from the same real footage, with clips reordered and trimmed or padded to match the report. Sampling replaces greedy decoding, so the chain is stochastic and the convergence argument roughly applies. The order in which candidate captions are listed is re-randomized at every generation, so that listing order cannot become an artificial attractor.

How the model reports depends on its type. For generative models such as VideoChat-T (OpenGVLab/TimeSuite), which already emits timestamps, we sample the grounded caption. For contrastive dual encoders of the kind TACT adapts, we score all $K!$ caption orderings for $K\le 4$ and sample one with probability $p(\pi)\propto \exp(s(V,\pi)/\tau)$. Here $\pi$ is an ordering, $s$ is the video-text similarity, and $\tau$ is a temperature calibrated on held-out data.

Measurements. We run many chains per video from varied starting orders and estimate the long-run ordering distribution $\hat q(\pi \mid \text{clips})$. We report how fast chains mix, measured as total variation distance to the pooled estimate across generations. We also characterize the attractor along four axes: script-likeness, measured by agreement with a text-only language model's order preference; how much content shapes it, measured by the change in $\hat q$ when clip contents are swapped but captions are kept; duration bias, measured as drift toward equal or stereotyped lengths; and sensitivity to listing order.

For each benchmark item we define prior surprise as $-\log \hat q(\pi^\ast)$, where $\pi^\ast$ is the true ordering. The first claim then becomes a specific prediction: prior surprise predicts order errors on held-out TempCompass event-order items and on Test of Time items.

An honest caveat applies. These models are not exact Bayesian samplers, so $\hat q$ is the chain's attractor, not a certified prior. The predictive tests do not require the stronger interpretation.

Intervention. We fine-tune a contrastive video-language model with a TACT-style loss. The hard negatives are high-$\hat q$ wrong orderings, meaning the orderings the model drifts toward, and training examples are weighted toward items with high prior surprise. If the attractor captures real failure structure, these negatives should carry more corrective signal than reversals do. The intervention reuses TACT's contrastive objective and changes only where the negatives come from.

Baselines

The naive reference point is the unmodified model. On the Test of Time ActivityNet order-consistency evaluation, foundation models score near chance; Test of Time: Instilling Video-Language Models with a Sense of Time reports 49.6%. On TempCompass, most video LLMs fall between 30% and 60%, with speed and direction at 25–33% (TempCompass).

The strongest existing method is TACT from the same Test of Time paper. It stitches clips together, uses time-reversed text and video as negatives, and reaches 85.7% order consistency on ActivityNet after 1–3 hours of training on four GPUs. We will re-run TACT with its released recipe on the same backbone, data, and number of GPU-hours as our method. The chains' inference cost will be reported separately and also charged against our budget in a matched-total-compute variant. Beating TACT means higher ActivityNet consistency than our reproduced TACT. On the TEMPO-style close-event setting, where Bagad et al. report TACT struggles, we need a gain beyond seed variance across three seeds. No reported result was found on Lacuna for TACT's close-event number, so our own reproduction is the reference there. Static recognition must also not degrade more than it does under TACT, which Bagad et al. note trades away spatial accuracy.

A stronger generic competitor is loss-based hard-negative mining. It picks, from all wrong permutations, the one the current model scores highest, and it uses the same budget.

Three ablations isolate the key ingredient. Uniform random permutation negatives test whether any extra negatives suffice. Negatives ranked by a text-only language model's order preference test whether the video chain adds anything beyond the language prior. Negatives taken from a single chain step test whether iterating matters.

For the diagnostic claim, the baselines are cheaper predictors of per-item errors: the model's one-shot confidence in the true order, the text-only order likelihood, and event spacing. We compare them by area under the ROC curve on held-out items.

To check that fine-tuning does not damage grounding, we report Charades-STA [email protected], using EaTR's 68.47% as a point of reference (Knowing Where to Focus: Event-aware Transformer for Video Grounding, jinhyunj/EaTR). The claim on that benchmark concerns no regression after fine-tuning. Beating EaTR is not the goal.

Why not just…?

Why not just read off the model's one-shot errors or confidence instead of running a chain? One generation of the chain is exactly this baseline, and it may be enough. The theory of regularization in transmission (Reali) predicts that weak biases grow over generations, so chain-derived surprise should predict errors better than one-shot confidence. We will plot predictive accuracy against generation count. If the curve is flat after the first generation, the chain adds nothing, and we will report that.

Why not just use a text-only language model's script prior? This is the most serious threat. Dynamic Reflections suggests the language side discards order, so video models may simply default to language stereotypes. The content-swap measurement and the text-prior negative ablation settle the question directly. If $\hat q$ matches the text prior and text-prior negatives match chain negatives, the finding narrows to "video-language order errors are inherited language priors." That is still a useful and falsifiable result, and the method would then be unnecessary.

Why not just mine hard negatives by current model loss? Loss mining finds orderings the model currently confuses at the moment of training. The chain finds orderings it drifts toward under repeated reproduction, which could be the same set. We run both at identical budget. If they tie, the chain earns its place only as a diagnostic tool, not as a training signal.

Why not just use TACT, or scale frames and captions at test time? TACT is strong on widely separated events, but it relies on abrupt cuts between stitched clips and loses spatial accuracy, which the head-to-head comparison above measures. Scaling frames and captions at test time raises overall alignment toward about 0.40 in Dynamic Reflections, but that paper also shows the order-blind behavior of language embeddings remains. We will check this by running chains under 8 and 48 frames: if the attractor is unchanged, more frames do not remove the prior.

Decisive Checks

The program succeeds as a diagnostic if three things hold: chains mix within a modest number of generations to a distribution that is clearly non-uniform and partly depends on clip content; chain-derived prior surprise beats one-shot confidence and the text-only prior at predicting held-out TempCompass and Test of Time order errors; and the attractor survives changes in frame count. It succeeds as an intervention if negatives taken from the attractor beat our reproduction of TACT, loss-mined negatives, and text-prior negatives on close-event order consistency, with ActivityNet consistency at or above 85.7%, and without a larger drop on Charades-STA or static retrieval. The likely failure modes are a degenerate attractor, a match with the text prior, or a tie with loss mining. Each one corresponds to a specific comparison above, so a negative outcome would still say where order blindness comes from.