Adding memory of past frames to a video gaze predictor should help it, and it usually does, but not in edited video. A model that predicts where people will look in a video usually does better when it can see the preceding frames, because motion and continuity tell it what viewers are tracking. In edited footage such as films, trailers and news, that advantage can reverse. A controlled study that added a spatiotemporal module to an otherwise identical model found a gain of 0.20 bits of information gain on smoothly shot video (LEDOV) and a loss of 0.07 bits on cut-heavy video (DIEM). The authors attribute the loss to the temporal window becoming a "semantically meaningless average" across hard cuts (Isolating the Role of Temporal Information in Video Saliency). The obvious fix is to detect the cut and throw away the memory. That fix also discards something useful. Edited video returns to earlier shots all the time: the two-person dialogue that alternates between speakers, or the anchor who reappears between news segments. The frames that best predict where viewers will look after a cut are often not the frames just before it, but frames from the last time the camera showed this view.
We propose to treat time the way edge-aware propagation methods treat image space. In edit propagation, a label or color spreads between pixels only when they are both nearby and similar in appearance, so propagation stops at real edges. We apply the same rule along the time axis. A frame draws context from a past frame in proportion to both their temporal distance and their visual similarity. A cut becomes an "edge in time" that blocks propagation, and a return to an earlier shot reconnects to that shot's memory. When nothing in memory matches, the model falls back to a static-plus-center-bias prior, which mimics the attentional reset viewers show after a cut. The claim is specific and testable. This similarity-gated memory should keep the LEDOV gain from temporal context, remove the DIEM loss, and beat a cut-detect-and-reset baseline specifically on frames where the video returns to a previously seen shot.
Where the idea comes from
The design borrows from affinity-based propagation, where pairwise terms weight label agreement by joint spatial and appearance closeness. Fully Automatic Video Colorization With Self-Regularization and Diversity uses a bilateral loss that forces pixels close in position and grayscale to share colors, with the penalty dropping at true edges. Bilateral Space Video Segmentation propagates labels in a six-dimensional grid over color, position and time. That paper treats time as a plain coordinate like x and y, and its authors note it breaks under fast motion. We keep the edge-preserving principle and move it to the level of whole frames and time. The appearance term, not the time coordinate, decides whether two moments are connected.
The temporal saliency side sets up the problem and one constraint. Going From Image to Video Saliency: Augmenting Image Salience With Dynamic Attentional Push models a scene-cut reset pathway that injects a Gaussian center-bias map. With its other pathways, it reports a Normalized Scanpath Saliency (NSS) of 2.21 on DIEM versus 1.43 for its base model. This is evidence that post-cut gaze behavior is structured, and we reuse the center-bias reset as our fallback prior. Unlike that work, we do not reset unconditionally: returning shots retrieve their memory. Existing long-range temporal modules, including the cross-frame attention in Weakly Supervised Video Salient Object Detection via Point Supervision and the adjacent-frame memory in Motion-aware Memory Network for Fast Video Salient Object Detection, were built for continuous, uncut clips. The adjacent-frame memory looks only at frames t−1 and t+1, so it cannot retrieve a shot from ten seconds earlier. Softmax attention always spends its full weight somewhere, so it cannot say "none of this is relevant." Unified Image and Video Saliency Modeling handles image versus video with a recurrent path it bypasses for static images. It has no mechanism for choosing which past frames count.
Core Mechanism
Take the UniFormerV2-based spatial model from the controlled study (UniformerSal-S) as the frame encoder. Give each frame $s$ a compact appearance descriptor $g_s$, trained so that frames from the same shot lie close together. Supervision comes from same-shot versus different-shot pairs, with shot labels from an automatic shot-boundary detector at training time only. Give each frame memory tokens $m_s$. Frame $t$ reads context
$$c_t = \frac{w_0\, p_0 + \sum_{s<t} k_{ts}\, m_s}{w_0 + \sum_{s<t} k_{ts}}, \qquad k_{ts} = \exp\!\left(-\frac{(t-s)^2}{2\sigma_\tau^2}\right)\exp\!\left(-\frac{\lVert g_t - g_s\rVert^2}{2\sigma_r^2}\right).$$
The symbols are defined as follows. $\sigma_\tau$ sets how far back in time memory reaches. $\sigma_r$ sets how similar two frames must look to be connected. $p_0$ is the static prediction combined with a learned center-bias map. $w_0$ is a learned fixed mass on this fallback.
The two range terms play the same roles as the spatial and color terms of a bilateral filter. A cut sharply drops the appearance term, so context collapses onto $p_0$. A return to an earlier shot restores a large appearance term despite the time gap, provided $\sigma_\tau$ is long. The fixed fallback mass is the key difference from softmax attention. When no memory matches, the output becomes the prior instead of an average over unrelated frames. The memory is a bounded bank of per-shot summary tokens, so cost grows with the number of distinct shots, not the number of frames.
Research Program
The work has three parts. First, build the gated memory on the UniformerSal-S/ST pair so that every comparison changes only the temporal module. Second, annotate shot structure on DIEM and Hollywood2 test sets and label each frame as within-shot, first appearance of a new shot, or return to a previously seen shot, along with time since the cut. This lets us attribute gains to the mechanism instead of reporting averages alone. Third, run a stratified analysis of time since the cut. Viewers reset toward center immediately after a cut, per the attentional push results, so we expect retrieved memory to add little in the first few frames after a return and to matter most in the following second, as viewers re-fixate on the previously attended speaker or object. If gains instead appear only in the first frames, the model is mostly learning a better reset prior, not using retrieval.
Baselines
The naive reference is UniformerSal-S applied frame by frame with no temporal context. The controlled study shows this already beats the spatiotemporal variant on DIEM by 0.07 bits information gain, so our first bar is to beat it on DIEM and not merely match it.
The strongest existing competitor is UniformerSal-ST from Isolating the Role of Temporal Information in Video Saliency. It reports +0.20 bits information gain over the spatial model on LEDOV and 1.053 bits on its metabenchmark, versus 0.520 for DeepGazeMR. It is the right comparison because it shares our backbone and training setup, and its only weakness is exactly the cut problem we target. We will retrain both models from the same initialization on the same training mixture, using the same number of tuning trials for learning rate and window length. Beating it means matching its LEDOV gain within noise while turning its DIEM deficit into a gain over the spatial model.
The ablation that most threatens the proposal is cut-detect-and-reset. Here UniformerSal-ST's window is truncated at automatically detected shot boundaries, and optionally the center-bias reset of Going From Image to Video Saliency is added after each cut. It gets the same detector our training uses, so detection quality is not a confound. Further ablations remove the fallback mass, which turns the memory into normalized attention; shrink $\sigma_\tau$ to the length of one shot, which keeps gating but removes cross-shot retrieval; and replace the learned descriptor with raw color histograms. For context we also report UNISAL (Unified Image and Video Saliency Modeling) and the attentional push model's DIEM NSS of 2.21. These are only reference points, since their backbones differ.
Why not just…?
Why not just detect cuts and reset the temporal window? This is the real threat, and on hard cuts alone it may match us. A reset removes the averaging artifact the controlled study identified. Where it must lose is on returning shots, because it discards the very memory that should predict the post-cut target. The decisive comparison is therefore on the "return to a previously seen shot" stratum of DIEM and Hollywood2. If reset matches us there too, the retrieval idea is wrong, and the honest contribution shrinks to a better-calibrated reset. We would report it that way.
Why not just use a longer attention memory and let the network learn what to attend to? Standard attention already weights frames by feature similarity, so in principle it could learn retrieval. But its weights must sum to one, which is how cross-attention summary tokens end up averaging across cuts. The long-range cross-frame attention in Weakly Supervised Video Salient Object Detection via Point Supervision was only tested on uncut benchmarks. Our no-fallback ablation is exactly this alternative with a long window, so the experiment settles it directly.
Why not just use the static model and skip temporal modeling on edited video? On DIEM this is currently the best choice. It gives up the LEDOV gain, though, and flicker and motion cues matter in continuous shots: temporal flow guidance lifts DAVIS max F-measure from 0.775 to 0.798 in Flow Guided Recurrent Neural Encoder. The proposal is only worthwhile if one model beats the static model on both datasets.
Proposed Experiments
We train on DHF1K, LEDOV and the Hollywood2 training split, and evaluate on LEDOV, DIEM, Hollywood2, UCF-Sport and DHF1K validation. Metrics are information gain, NSS, AUC and correlation coefficient, each broken down by our shot-structure strata and by time since the cut. The first decisive check is the two-dataset test. On LEDOV, we must match UniformerSal-ST's reported +0.20 bits gain over the spatial model. On DIEM, where the study reports a 0.07-bit deficit, we must beat the spatial model. The second decisive check is the returning-shot stratum against cut-and-reset. The third is the mechanism check. On clips with alternating shots, the kernel weights $k_{ts}$ should concentrate on frames of the matching earlier shot. We will measure this with the annotated shot labels.
As a supplementary probe we splice LEDOV clips into A-B-A sequences to test whether retrieval finds the right shot. Viewers never saw those spliced cuts, so their recorded gaze contains no post-cut reset. This probe tests retrieval only and does not count toward the gaze-prediction claims. No reported result was found on Lacuna for returning-shot strata on any dataset, so these numbers will be new.
References
- Isolating the Role of Temporal Information in Video Saliency: A Controlled Experimental Analysis
- Fully Automatic Video Colorization With Self-Regularization and Diversity.
- Bilateral Space Video Segmentation
- Going From Image to Video Saliency: Augmenting Image Salience With Dynamic Attentional Push
- Weakly Supervised Video Salient Object Detection via Point Supervision
- Motion-Aware Memory Network for Fast Video Salient Object Detection
- Unified Image and Video Saliency Modeling
- Flow Guided Recurrent Neural Encoder for Video Salient Object Detection