When a person wearing a head-mounted eye tracker finishes cutting a tomato and reaches for a pan, their eyes usually move to the next object before their hands do. That shift in gaze is a free, unlabeled marker of where one activity ends and the next begins. This note proposes a way to read such markers automatically and in real time. A model learns to predict the wearer's next gaze sample. In the model's internal representation, only a few units are allowed to be active at once, and they compete to be among those few. We place a boundary wherever the set of winning units changes abruptly. We do not place it where the raw prediction error spikes, because raw error spikes at every saccade. Saccades are the fast, ballistic eye jumps that occur several times per second within a single activity.
The opportunity matters for two reasons. First, the dominant self-supervised method for finding boundaries flags moments when a predictor of the future is most surprised. That method has been applied to video frames, speech and music, but never to the observer's own eye movements. Second, gaze has a poor track record as an extra input: adding it to learning models often makes them worse. Our bet is that both facts share one cause. Gaze is unpredictable at the scale of single saccades and predictable at the scale of what is being attended to. Competitive sparse coding gives a principled way to separate those two scales.
Core Claim
In egocentric (first-person) video with synchronized gaze, a causal gaze predictor whose hidden layer uses $k$-winner-take-all competition yields a boundary signal that is both more accurate and more robust across people than (a) raw gaze prediction error, (b) the same predictor with dense activations, and (c) video-only prediction error. Here $k$-winner-take-all, or $k$-WTA, means that only the $k$ most active units in a layer keep their values and the rest are set to zero. Our signal is code turnover, the fraction of winning units that change between consecutive time steps:
$$\tau_t = 1 - \frac{|S_t \cap S_{t-1}|}{k},$$
where $S_t$ is the set of $k$ winning hidden units at time $t$. Within an activity, saccades mostly re-weight units that are already winning, so $\tau_t$ stays low. When the attended task changes, a different subset of units wins and $\tau_t$ jumps. A secondary claim concerns combining this gaze signal with a video-based boundary signal without using labels. A maximum-entropy reweighting step, borrowed from integrative structural biology, does this better than averaging the two signals.
Relation to Prior Work
Prediction-error segmentation supplies the core recipe. A Perceptual Prediction Framework for Self Supervised Event Segmentation flags boundaries where predicted and observed video frames diverge. Generic Event Boundary Detection formalizes boundaries as local minima of predictability and benchmarks an unsupervised PredictAbility baseline. CoSeg uses errors in reconstructing masked features. Blind Phoneme Segmentation With Temporal Prediction Errors and Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors carry the same idea to sound. All of these predict the external stream. We predict the observer's gaze trajectory, where the naive error signal fails for a structural reason: saccades are intrinsically unpredictable. Online Generic Event Boundary Detection motivates our strictly causal setting, since most boundary detectors look at the whole video in both directions.
Eye-tracking work motivates the gap and also constrains the design. Scanpath Complexity shows that saccade statistics, scored against a two-mode "ideal reader" distribution, carry a strong signal of cognitive effort. It uses that signal as a static difficulty index rather than to segment a stream. Movie Editing and Cognitive Event Segmentation in Virtual Reality Video observes that discontinuities trigger a saccadic "search mode" with fewer stable fixations. It describes this behaviorally with a fitted exponential curve and builds no detector. Mutual Context Network for Jointly Estimating Egocentric Gaze and Action couples gaze and action on EGTEA. It is supervised by action classes and reports difficulty with exactly the anticipatory gaze shifts we treat as signal.
Two negative results constrain us. Adding eye-tracking AOI data to models of representation skills does not improve prediction accuracy found that gaze, binned into screen regions, added noise and made LSTMs overfit. Reading Task Classification Using EEG and Eye-Tracking Data found that accuracy collapsed to 52–57% when models were tested on people they had not been trained on. Any claim that gaze helps must therefore survive evaluation on held-out subjects and comparison against a no-gaze baseline.
Lateral inhibition supplies the mechanism. Compete to Compute shows that local winner-take-all blocks route different inputs through different sub-networks and reduce forgetting. Winner-Take-All Autoencoders shows that top-$k$ sparsity learns specialized features without fragile sparsity penalties. It also warns that too small a $k$ prevents the network from learning features that combine. Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations shows that $k$-WTA activations occupy low-rank subspaces per task. However, it receives task boundaries from outside. We invert that relationship: changes in which subspace is active become the evidence for a boundary. The hippocampal pattern-separation account of O'Reilly et al. gives the intuition that sparse codes turn overlapping inputs into non-overlapping ones, which is what makes turnover discrete.
Integrative structural modeling supplies the fusion method, not the application. Structure and dynamics of a nanodisc reweights a molecular-dynamics ensemble by Bayesian/maximum entropy so it agrees with sparse, heterogeneous measurements while staying as close as possible to the prior. The reviews by Heller et al. and Bayesian methods in integrative structure modeling frame the general weighted-restraint objective. Gauto et al. shows the payoff of combining a global low-resolution envelope from cryo-EM with local precise anchors from NMR. We use the same division of labor. Video prediction error gives a dense, smooth, low-precision boundary prior; this is the envelope. Gaze turnover gives sparse, sharp anchors.
Research Program
The gaze predictor. Each time step's input is a short window of gaze coordinates and velocities, optionally concatenated with a small feature patch from the video around the gaze point. A causal recurrent or temporal-convolution encoder maps this to a hidden layer with $k$-WTA. To avoid dead units, we add the usage-frequency dropout of the null-space paper. A decoder then predicts the next gaze window. Training is purely self-supervised on gaze, with no action labels. We sweep $k$ to locate the factorization-collapse regime that the Winner-Take-All Autoencoders paper warns about.
Boundary readout. We compute $\tau_t$ online, smooth it with a short causal filter, and pick peaks with a single threshold. The threshold is tuned on held-out validation subjects.
Unsupervised fusion. A video-only causal predictor yields a boundary probability $p_t$. We reweight it to $q_t \propto p_t\, e^{\lambda \tau_t}$. Following BME, $\lambda$ is chosen as the smallest tilt that makes the expected gaze turnover at predicted boundaries match a target estimated from turnover statistics. This keeps the reweighting maximally close to the video prior. The fusion stays label-free. We acknowledge that a single-constraint maximum-entropy reweighting is mathematically an exponential tilt, which is log-linear fusion. The test, set up below, is whether the constraint-based choice of $\lambda$ matches a label-tuned weight.
Analysis. We check whether high-turnover moments precede action onsets, as the anticipatory-gaze account predicts. We also check whether turnover peaks coincide with the drop in stable fixations reported in the VR editing study.
Baselines
The naive reference is uniform boundary placement at the average action length, together with a heuristic that marks a boundary at any saccade above an amplitude threshold. Neither involves learning. If our method does not clearly beat the saccade heuristic, gaze turnover is just a saccade detector.
The second baseline is raw gaze prediction error from a dense predictor with the same architecture and parameter count. It is trained with the same data, steps and tuning budget, and given the same causal smoothing and threshold search. This is the direct transplant of prediction-error segmentation onto gaze.
The key ablation isolates competition from mere sparsity and smoothing. It uses the identical predictor with dense ReLU units plus an L1 activation penalty tuned to match the average number of active units. Turnover is then computed on its top-$k$ units after the fact. A second variant uses local winner-take-all blocks as in Compete to Compute. If the post-hoc top-$k$ readout on a dense network performs as well, the trained competition is unnecessary and we will say so.
The strongest unsupervised competitor is video-only prediction error in the style of the PredictAbility baseline, rerun causally on EGTEA with the same backbone features. The Generic Event Boundary Detection paper reports PredictAbility at F1 0.527 on TAPOS and a supervised pairwise classifier at F1 0.817 on Kinetics-GEBD. There is no reported result on Lacuna for boundary detection on EGTEA, so we will reproduce these baselines ourselves. Beating this baseline means a statistically significant F1 gain from fused gaze+video over video-only, averaged across leave-subject-out folds.
As a supervised ceiling, we derive boundaries from changes in the frame-wise predictions of an action recognizer like the Mutual Context Network. That network reports 62.6% action accuracy on EGTEA. For fusion, the competitor is logistic late fusion of $p_t$ and $\tau_t$ with a weight tuned on labels.
Why not just…?
Why not just detect large saccades or fixation-rate drops? The VR editing study suggests that search mode is visible in fixation statistics alone, so this heuristic is a real threat at hard cuts. In continuous kitchen activity, however, large saccades occur constantly within one action, for example when checking a pot or reaching for an ingredient. We expect the heuristic to have high recall and low precision. The amplitude-threshold baseline settles the question directly, and we will report precision and recall separately.
Why not just use video prediction error and ignore gaze? Given the overfitting result of Rau et al. and the cross-subject collapse in reading-task classification, this is the most likely way the proposal fails. Our answer is empirical. The video-only and fused systems share features, folds and tuning budget, and fusion must beat video-only on held-out subjects. If it does not, the paper reports a sharper negative result than Rau et al.: one using continuous gaze rather than screen-region bins.
Why not just smooth the raw gaze prediction error, or add an L1 penalty? Smoothing removes saccade spikes but also blurs boundaries in time. An L1 penalty gives sparsity without forcing units to compete. The matched-sparsity L1 ablation and the smoothed dense baseline test whether trained competition matters. We predict the gap will be largest under tight localization tolerances.
Why not just learn the fusion weight with labels? That works but gives up the unsupervised claim. If the maximum-entropy choice of $\lambda$ falls within noise of the label-tuned weight, the borrowed structural-biology method earns its place. If it does not, we keep the turnover signal and drop the fusion claim.
Proposed Experiments
The main testbed is EGTEA, with 29 hours of egocentric kitchen video, synchronized gaze, and 106 action classes. Boundaries come from the starts and ends of action segments. We score with the F1 at relative-distance tolerances used by Kinetics-GEBD, sweeping the tolerance from 0.05 to 0.5 of segment length. All results use leave-subject-out folds; we report within-subject results only to quantify the generalization gap. For TAPOS and Kinetics-GEBD, which lack gaze, we test transfer by feeding gaze predicted by an EGTEA-trained gaze model. This is an exploratory check of whether predicted gaze retains the turnover signal. If the 360° VR dataset from the editing study can be obtained, it offers a second, cleaner check. There, human-marked boundaries (59% for action discontinuities versus 16% for continuity cuts) give graded ground truth against which $\tau_t$ can be correlated.
The decisive checks are three. First, on held-out subjects, competitive turnover must beat both raw gaze prediction error and the matched-sparsity dense ablation; otherwise the lateral-inhibition claim fails. Second, fused gaze+video must beat causal video-only prediction error; otherwise the gaze claim fails. Third, peaks in $\tau_t$ should lead action onsets by a consistent positive lag. That lag would show the signal reflects anticipatory attention rather than motion artifacts that follow the action.
References
- A Perceptual Prediction Framework for Self Supervised Event Segmentation
- Generic Event Boundary Detection: A Benchmark for Event Segmentation
- CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation
- Blind Phoneme Segmentation With Temporal Prediction Errors
- Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors
- Online Generic Event Boundary Detection
- Scanpath Complexity: Modeling Reading Effort Using Gaze Information
- Movie editing and cognitive event segmentation in virtual reality video
- Mutual Context Network for Jointly Estimating Egocentric Gaze and Action
- Adding eye-tracking AOI data to models of representation skills does not improve prediction accuracy
- Reading Task Classification Using EEG and Eye-Tracking Data
- Compete to Compute
- Winner-Take-All Autoencoders
- Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations
- A Hippocampal Model of Recognition Memory
- Structure and dynamics of a nanodisc by integrating NMR, SAXS and SANS experiments with molecular dynamics simulations
- Methods of probing the interactions between small molecules and disordered proteins
- Bayesian methods in integrative structure modeling
- Integrated NMR and cryo-EM atomic-resolution structure determination of a half-megadalton enzyme complex