Knowing When to Trust a Prior: Reliability-Gated Cue Fusion for Video Gaze Prediction

Lichen Zhu, Yueqian Lin, Yiheng Wang, Hai "Helen" Li, Yiran ChenarXiv 2026 · arXiv

TL;DR

FocusGate is a gated ensemble of "gaze-free" priors (like defocus, motion, and face detection) that improves video gaze prediction by knowing exactly when to trust its inputs. By using a per-frame reliability gate to filter out noisy defocus cues, it consistently boosts the performance of state-of-the-art supervised models across multiple domains with negligible latency overhead.

What Problem Does This Paper Address?

Where do people look when watching a video? While modern video saliency is dominated by massive, gaze-trained deep networks, there is still valuable signal in "gaze-free" priors—explicit cues like what’s in focus, where faces are, or how objects move.

The problem is that these priors aren't always reliable. A defocus map is a great predictor of gaze in a shallow depth-of-field shot (where the subject is sharp and the background is blurred), but in a deep-focus shot, the same estimator might just output texture noise. If you always fuse these priors into your model (unconditional fusion), the noise from unreliable frames can actually hurt overall performance. The authors set out to build a system that can "abstain" from making a prediction when a specific cue is likely to be wrong.

Methods

The authors introduce FocusGate, an ensemble of eight gaze-free priors (including center bias, face detection, motion, and DINO-based attention).

The core innovation is the Reliability Gate ($c_t$). Instead of looking at the ground truth (gaze data), the gate analyzes the shape of the defocus map ($D$) itself. It calculates three statistics:

  1. Spread: The difference between high and low probability values.
  2. Separation: How distinct the "in-focus" and "out-of-focus" modes are.
  3. Sharp Fraction: The percentage of the frame considered in-focus.

If the map is "bimodal" (clear subject vs. clear background), the gate opens. If it's "unimodal" (everything is sharp or the map is just noise), the gate closes, and the prior effectively abstains.

Fig. 1. FocusGate. (a) Eight gaze-free priors (fitted weights wk) are rank-summed into the base B. The defocus map D enters only through the gate ct, which thresholds three shape statistics of D and never sees the fixations. (b) The gate opens on a bimodal D and closes on a unimodal one, where the prediction reduces to B exactly.

To handle these abstentions without adding complex parameters, they use midrank normalization. This ensures that if a prior (like a face detector) finds nothing, it produces a constant map that remains inert during the fusion process. FocusGate can be used as a standalone predictor or added to a supervised model ($M$) by simply summing their rank-normalized outputs.

Datasets

The researchers evaluated FocusGate across four distinct domains to ensure robustness:

  • Hollywood-2 (H2): Professional film footage (primary test set).
  • UCF-Sports: Action-heavy sports clips (transfer domain).
  • DHF1K: Web-style videos (transfer domain).
  • DIEM: A confirmatory fourth domain, pre-registered and scored only once after all parameters were frozen to prevent over-fitting.

The defocus estimators and gate thresholds were tuned using the CUHK blur detection dataset, entirely independent of any gaze-tracking data.

Key Insights and Ideas

  • The Collapse Regime: The authors discovered that defocus estimators have a specific "collapse regime" where their performance drops to chance. Crucially, this regime is identifiable purely from the estimator's own output (see Figure 2). Gating works because it identifies these "chance" frames and discards them.
  • Complementarity over Strength: Even though FocusGate alone isn't as strong as the latest 6.9-billion-parameter supervised models, it still improves them. This proves that gaze-free priors contain "residual" signals that even the largest models haven't fully absorbed.
  • Efficiency: Because the gate relies on simple shape statistics, it is incredibly fast. Adding FocusGate to the NTIRE 2026 champion model (ViSAGE) adds only 1% to its total latency.
Fig. 2. The collapse regime. Standalone sAUC of each cue on the 1,715 test frames by quintile of the spread of its own map (shaded: 95% CI). Defocus maps sit at chance when uni- modal and rise when bimodal, while sharpness and motion never approach chance.

Key Results

FocusGate demonstrated impressive consistency across all tests:

  • Universal Improvement: It improved the shuffled AUC (sAUC) of four different supervised models across all sixteen model-domain combinations tested.
  • Converting Harm to Gain: On sports footage, where the defocus cue is particularly noisy, unconditional fusion lowered performance by 1.40 points. The reliability gate flipped this into a significant gain of 0.68 points.
  • State-of-the-Art Boost: When added to ViSAGE, it achieved the highest scores in the study (e.g., 81.9 sAUC on film).
  • Standalone Power: With a 16-frame causal mean, the ensemble alone outperformed older supervised models like TASED-Net and UNISAL on film and DIEM datasets.
Table 1. sAUC (%) of every predictor under one protocol on held-out video (Sec. 3). ms/frame per component on one A100 (batch 1, one forward pass per predicted frame, two for ViSAGE’s expert pair). A “+ FocusGate” row adds 111 ms to its model. Bold with ∗: paired 95% cluster CI vs. the model’s own row excludes zero and survives a Holm correction over the sixteen cells. ‡: confirmatory domain, pre-registered and scored once (the ViNet-S cell was added afterwards and is exploratory). These cells measure the ensemble, not the gate, whose pre-specified contrast is null (Sec. 4.2).

Limitations and Future Work

The current reliability gate is highly specialized for defocus cues. The authors found that other cues, like motion or general sharpness, didn't exhibit the same clear "collapse to chance" behavior, meaning the current gating logic didn't provide the same benefits for them.

Future research will focus on identifying similar reliability statistics for other types of priors. Additionally, while FocusGate excels at ranking (sAUC), it doesn't inherently improve density-based metrics (like NSS) without a separate calibration step, as the priors primarily provide information about the order of importance rather than the absolute probability density of fixations.

Summary updated

Abstract

Video gaze prediction is led by gaze-trained models, yet gaze-free priors carry signal those models have not absorbed, if one knows when to trust them. We propose FocusGate, a gated ensemble of gaze-free priors whose members may abstain. A per-frame gate reads three shape statistics of a defocus map and selects the frames on which the estimator is above chance on average, so rejected frames reduce to the base exactly, while midrank normalisation lets an all-zero prior abstain at zero parameters. Gated fusion is significantly positive on film, sports and web video, whereas unconditional fusion is harmful on sports and null on web. Added to four supervised predictors, the NTIRE 2026 champion among them, FocusGate improves all sixteen model-domain cells in shuffled AUC, fifteen significantly, one domain pre-registered and scored once, while adding only 1% to the champion's latency. Alone, it surpasses TASED-Net and UNISAL in shuffled AUC on film with a 16-frame causal mean.
BibTeX
@misc{zhu2026knowing,
  author = {Lichen Zhu and Yueqian Lin and Yiheng Wang and Hai "Helen" Li and Yiran Chen},
  title = {{Knowing When to Trust a Prior: Reliability-Gated Cue Fusion for Video Gaze Prediction}},
  year = {2026},
  eprint = {2610.08663},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.08663},
}
Figure