Watching Eyes as Physics Judges: Anticipatory Gaze as a Metric for Video Physics

People watching an object fall or roll do not wait for it to arrive. If their eyes run ahead to where they expect it to go, then a video that breaks physics should leave a visible trace in the viewer's eye movements. The eyes land where the object should be, the object is somewhere else, and the eyes jump to correct. This proposal turns that trace into an evaluation metric for generated video. We would record viewers' eye movements on real and generated physical events. We would then train a model to predict where a viewer's eyes will go next on real footage. A generated clip would be scored by how badly it surprises that model of human anticipation. The aim is a physics score that agrees with what humans actually notice, at the timescale where they notice it. It should also be computable without new humans once the model is trained.

The need is concrete. Video generators produce sharp frames with broken mechanics, and the standard scores do not catch this. Fréchet Video Distance, which compares feature statistics of real and generated clips, is dominated by single-frame appearance and misses frame shuffling and motion jitter (STREAM). Multimodal language models used as judges favour visually striking clips and explain away errors as rare real events (PhyWorldBench). Physics-specific scores such as trajectory error against a reference (PISA Experiments) penalise any deviation from one recorded outcome, including plausible alternatives (Naturalistic Head Motion Generation from Speech). None of these measures where a human observer's expectation breaks.

Core Claim

Anticipatory gaze, meaning where a viewer looks a fraction of a second to a few seconds ahead of the action, records the viewer's moment-by-moment physical expectation. The size of the mismatch between that expectation and what the video shows should predict human judgments of physical implausibility better than appearance-based, spectral, or language-model metrics. The mismatch should also locate the violation in time and space.

A second claim is scientific rather than technical. By comparing gaze-measured surprise against ground-truth physics, we can map which violations humans perceive and which they do not. For example, persistent sub-Earth gravity, which Objects in Generated Videos Are Slower Than They Appear documents in generators, may go unnoticed. A few-frame interpenetration at contact may be noticed immediately. This separates "perceptually wrong" from "physically wrong," a distinction the current literature blurs.

Where this sits relative to prior work

The gaze-anticipation literature motivates the mechanism. It shows that gaze leads the body at several timescales. Eyes lead the head by about 0.12 s in 360-degree viewing (Where Are You Looking?). Fixations precede object contact by about 4 s in unscripted kitchen activity (HD-EPIC). Gaze resolves the goal earlier than joystick input does (Gaze Complements Control Input).

Every one of these works uses gaze to predict what the person looking will do. GIMO predicts body motion. Gaze-Guided GNN predicts intention and atomic actions. EgoGazeVQA predicts intent answers from multimodal models. We flip the direction. We use an observer's gaze as a readout of their prediction about the world they watch, and we score the video against that prediction. The transfer from "actor's gaze leads actor's action" to "observer's gaze leads observed event" is a hypothesis this project tests first. It is not something the file establishes.

The physics-evaluation work constrains the design. VACT and PhyWorldBench show that generators are "consistently wrong" and that language-model judges need adversarial prompting to stay honest. Both evaluate whole clips with a verdict and give no timing. STREAM separates temporal from spatial quality through a frequency analysis of frame features. That analysis is statistical and has no notion of where a viewer expects an object to go.

VideoREPA supplies a method. It shows that a small video encoder, VideoMAEv2 at 86M parameters, carries more physical understanding than a 2B-parameter generator. We reuse that encoder as the backbone of our gaze predictor and as a no-gaze surprise baseline.

The temporal-scale literature supplies the second design constraint. A single fixed window either averages away brief events or misses long context (Benmokhtar). Dual-rate pathways handle both (HST-LSTM). Physics violations come in both kinds, a two-frame teleport and a three-second drift, so the metric must read anticipation at several horizons.

Research Program

The first stage is data collection. Participants view clips on a screen-based eye tracker, the 1200 Hz class used by Özdel et al..

The stimulus pool has three parts. The first is real physical events: the 361 real 120-fps drop videos of PisaBench (PISA Experiments) plus rendered collision scenes of the CATER type used by STREAM.

The second part is controlled corruptions of those clips. We inject violations of known type, onset and duration: teleportation, interpenetration at contact, gravity rescaled by a factor from 0.3 to 1.5, sudden mass or shape change, and local time reversal. Durations span from two frames to three seconds. Because these violations are injected and labelled, they give exact ground truth for both physics and timing.

The third part is generated clips from open generators on PhyWorldBench- and PisaBench-style prompts. These include the physics-post-trained models (PISA, VideoREPA) and their base models.

After each clip, viewers give a plausibility rating and a timestamp click at the moment they first noticed something wrong. This yields three aligned signals per clip: continuous gaze, a scalar rating, and a reported onset.

The second stage is the analysis that tests the premise. On real clips, we measure whether gaze leads the object, and by how much, across event types. On corrupted clips, we measure whether violations produce a stereotyped response: a corrective jump of the eyes within a few hundred milliseconds, or a growing smooth-pursuit lag for slow drifts. We also measure whether the size of that response tracks violation size and duration. This produces the perceptibility map described in the Core Claim, tied to violation type and timescale.

The third stage is the metric. We train a gaze-anticipation model on real clips only. The model uses a VideoMAEv2 encoder with two temporal pathways in the style of HST-LSTM: a fine one at full frame rate and a coarse one over multi-second context. Given frames up to time $t$, it predicts a distribution over the viewer's gaze location at $t+\tau$ for horizons $\tau$ from 0.1 s to 2 s.

On a test clip, the score is the model's surprise at what the video actually shows, at each horizon:

$$S(v) = \sum_{\tau} w_\tau \, \frac{1}{T}\sum_{t} -\log p_\theta\big(o_{t+\tau} \mid v_{\le t}\big),$$

The symbols are defined as follows. $v_{\le t}$ is the clip up to time $t$. $o_{t+\tau}$ is the location of the attended object at $t+\tau$, taken from an off-the-shelf tracker. $p_\theta$ is the predicted gaze distribution. $w_\tau$ are per-horizon weights fit on held-out ratings.

The per-time, per-horizon terms localise the violation. The horizon at which surprise peaks indicates the violation's timescale. Because the model is trained only on real footage, no generated clip enters training. This guards against the metric learning a particular generator's artifacts.

Baselines

The naive reference point is trajectory L2 against the source clip, the PisaBench metric. On real-world PisaBench the PISA model reports 0.067 against 0.174 for Sora and 0.187 for Runway Gen3 (PISA Experiments). A frame-level score, PSNR, sits alongside it. These numbers are model scores, not evaluator agreement. No reported result on Lacuna was found for how well any of these metrics correlates with human plausibility ratings, so all agreement figures will be measured fresh on our set.

The distribution-level baselines are Fréchet Video Distance and STREAM-T (STREAM). FVD is the most widely used score. STREAM-T is the strongest temporal statistic in the file. Both are run on identical clip sets using their published encoders.

The strongest existing evaluator is a multimodal language-model judge with the Context-Aware Prompting of PhyWorldBench. That prompting was designed specifically to remove aesthetic bias and error rationalisation, which makes it the direct competitor on "does this look physically wrong." We run it with the same open model at the same frame sampling, and tune its prompt on the same held-out split used to fit our $w_\tau$. The paper reports only generator success rates, with Pika near 26%, and not judge-human agreement. No reported result on Lacuna was found for CAP judge correlation with human ratings.

The key ablation is a no-gaze surprise model. It has the same VideoMAEv2 encoder, the same two pathways, and the same real training clips. The difference is that it predicts the object's future position directly instead of the viewer's future gaze. This isolates whether human anticipation adds anything beyond a learned physics predictor.

A second ablation fits a regressor on VideoMAEv2 features to plausibility ratings. It uses an annotation budget equal to our gaze budget, measured in participant-minutes. A third ablation collapses the predictor to a single 0.5 s horizon, which tests the multi-scale design.

Beating the field means the following. Spearman correlation with mean human ratings on held-out generated clips must exceed both the CAP judge and the no-gaze ablation, with non-overlapping bootstrap intervals. Area under the ROC curve for detecting injected violations must be higher, especially for violations shorter than 200 ms. Onset localisation error against viewers' clicked timestamps must be lower.

Why not just…?

Why not just use a better-prompted multimodal judge? Context-Aware Prompting already reduces aesthetic bias (PhyWorldBench), and it may match us on clip-level correlation. Its weak point is short events. Judges see sparsely sampled frames, so a two-frame interpenetration may never be shown to them. The injected-violation set, stratified by duration, tests this directly. If the judge matches us at all durations, the claim reduces to localisation, and we will report it that way.

Why not just measure surprise with a physics predictor, no humans? This is the most serious threat. VideoMAEv2 already carries physical knowledge (VideoREPA). The no-gaze ablation is this alternative, built on the same backbone and the same data. Our expectation is that it will flag violations humans never perceive, such as mild gravity rescaling, and so correlate worse with ratings. If it matches gaze-trained surprise on ratings, then gaze is not earning its cost. In that case we would keep only the perceptibility map as the contribution.

Why not just collect ratings and train a regressor? Ratings are cheaper than eye tracking. They give one number per clip, with no timing and no location. The equal-budget regressor ablation settles which signal is more data-efficient. Gaze must at least match ratings on correlation and win on localisation to justify itself.

Proposed Experiments

The first decisive check is the premise. On real PisaBench and collision clips, viewers' gaze should lead the object at a consistent horizon. On injected violations, corrective eye jumps should appear more often than on matched uncorrupted clips. If neither effect appears, the project stops at a negative result about observer gaze, which is still worth reporting.

The second check is the metric comparison described above. It runs on held-out generated clips from base and physics-tuned generators, using correlation, detection, and localisation, against every baseline.

The third check is the perceptibility map. For each violation type and duration, we compare the gaze-detection rate with the ground-truth physical error. We report where they diverge, and whether physics post-training (PISA, VideoREPA) fixes perceived or unperceived errors.

The fourth check extends to agent-driven scenes. We would show HD-EPIC clips and generated egocentric futures of the kind produced by EVA, where the actor reaches for an object other than the one gaze has signalled. This tests whether the same surprise signal catches broken intent, a failure EVA itself leaves to GPT-4o scoring.