People looking at the same picture do not move their eyes the same way. Some scan the background, some lock onto faces, some search methodically. How much people agree also depends on the image: some scenes pull everyone to the same spot, while others scatter observers. Current probabilistic gaze models are generators that sample many plausible sequences of eye fixations (scanpaths) for one image. They capture this variety only implicitly, inside a single shared network. This proposal makes it explicit. We build a scanpath diffusion model, meaning a generator that learns to turn random noise into realistic fixation sequences step by step. Its denoising network contains a small set of "expert" sub-networks, and each sampled scanpath is handled by one expert, chosen once for the whole trajectory. The choice of expert is treated as a hidden variable, a discrete viewing strategy. An image-conditioned prior predicts which strategies are likely for a given scene. A posterior, computed from how well each expert explains a real human scanpath, says which strategy a given observer actually used.
In two sentences: the probabilities that decide which expert handles an input become a learned distribution over human viewing strategies, predicted from the image and inferred from an observer's past scanpaths. Prior mixture-of-experts work routes individual tokens. That kind of routing is known to specialize by denoising timestep and network depth rather than by meaning. Routing whole trajectories is what could make the experts line up with how people actually look. If it works, one model would supply three things. The entropy of its prior over strategies would predict how much observers disagree on each image. Its posterior would identify an observer's style from a few support scanpaths. Its experts would separate groups such as autistic and neurotypical viewers without being given their labels.
Core Claim
A trajectory-level, latent-variable mixture of denoising experts, anchored by a shared expert, will match or beat dense scanpath diffusion on standard similarity metrics. It will also do three things that dense models with embedding-based conditioning do not provide as directly. It will cover inter-observer diversity better. Its image-level routing entropy will predict human inter-observer consistency. Its few-shot posterior over experts will personalize scanpaths at least as well as learned observer embeddings.
The claim is falsifiable at one specific point. A dense model that receives the same discrete strategy code through feature modulation, with matched active parameters, may do just as well. If it does, the gain comes from the discrete latent and not from separating parameters, and the proposal should be reported that way.
Grounding in Prior Work
Two strands of work motivate the idea.
The first is probabilistic gaze modeling. ScanDiff generates discrete scanpaths with a diffusion model conditioned on Vision Transformer image features and CLIP text embeddings of the task, covering both free-viewing and visual search. DiffEye generates raw continuous 240Hz eye-tracking trajectories from DINOv2 features. Both use dense backbones in which every sample, task, and timestep passes through the same weights. Their diversity comes only from the sampling noise, so neither offers a handle on which strategy produced a given sample. We reuse ScanDiff's conditioning and output format as our backbone and change only the denoiser.
Predicting When Saliency Maps are Accurate and Eye Fixations Consistent showed that inter-observer consistency is a predictable property of an image, driven largely by its object categories. That result supplies a target our prior's entropy should track. Pathformer3D predicts a Gaussian mixture over fixation locations at each step from a dense transformer. Its mixture lives at the output and resets every step, whereas ours is a single persistent choice per trajectory, made inside the network.
The second strand is individualized gaze modeling, which motivates the personalization test. Beyond Average and Few-shot Personalized Scanpath Prediction encode each observer as a continuous "gaze fingerprint" vector and use it to modulate a dense predictor. They report strong few-shot and autism-related results, so they are the main competitors. We replace the fingerprint with a posterior over discrete experts and test whether that does better.
Three papers constrain the design.
DiT-MoE scales diffusion transformers with sparse token-level experts plus a shared expert. Its analysis found that experts specialize by timestep and layer depth, not by semantic class. We read this as evidence that per-token routing will not produce strategy-level experts, which is why we route per trajectory.
Imitation Learning from MPC for Quadrupedal Multi-Gait Control found that standard losses gave only 40% single-expert responsibility for a gait. A competitive log-partitioned loss raised this to 100%. We borrow the competitive-responsibility idea to prevent routing collapse.
Certain Head, Uncertain Tail treats routing as something to sample from at test time. We go further and make sampling from the router the main source of strategy-level diversity.
Two papers in the file point the opposite way. CogMoE uses gaze as an input for predicting cognitive load and does not predict gaze. CARE-Edit routes by editing condition and does not involve gaze.
Research Program
The backbone is a ScanDiff-style denoiser. Each transformer block keeps its shared feed-forward expert and adds $K$ small routed feed-forward experts, with $K$ between 4 and 8. Each expert is much smaller than the shared one, because gaze datasets are small. COCO-Search18, for example, has roughly 300,000 fixations in total, so large separate experts would overfit.
For a training pair, $c$ is the image and task and $s$ is a human scanpath. A prior network $\pi_\phi(k \mid c)$ assigns a probability to each expert $k$. During training, every expert computes its denoising loss $L_k(s,c)$ at the same sampled noise level, which is cheap because only the expert layers are duplicated. These losses define each expert's responsibility $r_k$ for that scanpath. The denoising loss stands in for a log-likelihood, and the temperature $\tau$ is tuned on validation data:
$$r_k(s,c) = \frac{\pi_\phi(k\mid c)\,\exp(-L_k(s,c)/\tau)}{\sum_j \pi_\phi(j\mid c)\,\exp(-L_j(s,c)/\tau)}.$$
The denoiser is trained on $\sum_k r_k L_k$, with $r_k$ held fixed. The prior is trained to match the responsibilities through the cross-entropy $-\sum_k r_k \log \pi_\phi(k \mid c)$. A small penalty keeps the dataset-average prior close to uniform, which prevents collapse onto one expert. Together these updates amount to soft expectation-maximization over a mixture of diffusion models, with experts competing for responsibility.
At test time there are three modes. Unconditional generation samples $k \sim \pi_\phi$ and then denoises. Personalization pools the responsibilities over an observer's few support scanpaths and samples experts from that pooled posterior. The entropy of $\pi_\phi(\cdot \mid c)$ is read off as the model's predicted inter-observer dispersion for the image.
An optional guided variant adds the known free-viewing-versus-search label as weak supervision on the responsibilities, in the spirit of the quadruped paper's guided loss. Autism labels are never used in training.
Baselines
The naive reference point uses no learned model of strategy. It samples human scanpaths from other images under the same task, and it also reproduces the human inter-observer ceiling by leaving one subject out. This bounds what any model can achieve and catches metrics that reward center bias.
The strongest existing generator is ScanDiff, retrained by us with the same features, data splits, and training budget. It is the strongest competitor because it already unifies free-viewing and search, and it already produces diverse samples. Our model must beat it on MultiMatch, ScanMatch, the semantic sequence score, and ScanDiff's own diversity metrics. The file reports no numbers for these: no reported result found on Lacuna for ScanDiff's MM, SM, SS, or SemSS values. We therefore rely on our own reproduction.
For personalization, the baseline is ISP-SENet, which reports a ScanMatch of 0.368 on OSIE in the 1-shot setting (Few-shot Personalized Scanpath Prediction). For unsupervised autism-versus-neurotypical separation, the baseline is ISP's clustered observer features, which reach up to 82.1% accuracy on OSIE-ASD (Beyond Average). Beating these means exceeding 0.368 one-shot ScanMatch on OSIE and exceeding 82.1% when clustering observers by their mean expert posteriors. Both comparisons use the same support-set sizes and the same splits.
Three ablations isolate the key ingredient, each with matched active and total parameters and identical tuning budgets. The first is the decisive one: a dense ScanDiff that receives the same $K$-way strategy code, inferred by the same expectation-maximization procedure, through feature modulation instead of separate experts. The second is a dense ScanDiff whose width is increased to match our total parameter count. The third is a DiT-MoE-style token-level router at the same expert count.
Why not just…?
Why not just condition a dense network on an observer embedding, as ISP-SENet does? This is a serious competitor, and its reported numbers are the bar we must clear. Our test is direct: same support sets, same splits, and comparison against the 0.368 one-shot ScanMatch and the 82.1% clustering accuracy. If continuous fingerprints win, the conclusion is that discrete strategies are too coarse for individual observers. Our model could still be useful for image-level dispersion, but we would not claim to improve personalization.
Why not just feed a discrete strategy code into a dense model? This is the biggest threat. A categorical latent passed in as an embedding is already a mixture model, and on small gaze datasets separate parameters may simply overfit. The matched code-through-modulation ablation decides the question. We will report it as the headline comparison, not as a supporting detail.
Why not rely on ScanDiff's sampling noise for diversity? Noise-driven diversity can match human spread on average, yet still fail on a per-image basis. It also gives no explicit quantity that can be checked against measured human consistency. Two checks separate the models: per-image diversity agreement, and the correlation between predicted dispersion and measured inter-observer consistency. For ScanDiff we measure dispersion as the spread of its samples.
Why not route per token, as DiT-MoE does? DiT-MoE's own analysis found that token-level experts specialize by timestep and depth rather than by meaning (Scaling Diffusion Transformers to 16 Billion Parameters). We still run the token-level ablation. We expect its experts to show weaker alignment with known observer groups.
Proposed Experiments
We train jointly on COCO-Search18 and COCO-FreeView. MIT1003 serves as a further free-viewing benchmark, and OSIE and OSIE-ASD serve as held-out tests of generalization and individual differences.
The first decisive check is the dispersion test. Across held-out images, the entropy of the prior over experts should correlate positively with the dispersion of human fixations, and the correlation should be stronger than that obtained from the spread of dense-model samples.
The second check concerns search efficiency on COCO-Search18. We plot the cumulative probability of fixating the target against fixation number, for both humans and the model. Humans reach about 50% on the first saccade and 94% by the third, while the foveated inverse-reinforcement-learning model plateaus around 83% (COCO-Search18). We report how close each model comes to the human curve. We do not expect routing alone to close this gap, and we state that limitation openly.
The third check is the unsupervised autism separation described under Baselines, using observer clusters formed from expert posteriors.
The final check is collapse diagnostics. We report expert usage, the per-image routing entropy, and responsibility sharpness, compared against the quadruped paper's 40% versus 100% single-responsibility contrast.
A positive outcome requires two things: parity or better on the standard similarity metrics, and a win over the discrete-code-modulation ablation on at least the dispersion and autism-separation tests. Anything less means the parameter separation is not what helps.
References
- Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction
- DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images
- Predicting When Saliency Maps are Accurate and Eye Fixations Consistent
- Pathformer3D: A 3D Scanpath Transformer for $360^{\circ }$ Images
- Beyond Average: Individualized Visual Scanpath Prediction
- Few-shot Personalized Scanpath Prediction
- Scaling diffusion transformers to 16 billion parameters
- Imitation Learning from MPC for Quadrupedal Multi-Gait Control
- Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE
- CogMoE: Signal-Quality–Guided Multimodal MoE for Cognitive Load Prediction
- CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing
- COCO-Search18 fixation dataset for predicting goal-directed attention control