Do Vision Mixture-of-Experts Routers Keep What People Remember?

Some vision models are built as a mixture of experts: many small sub-networks ("experts"), plus a learned gate that sends each image patch to only one or two of them. These models already make a quiet, consequential choice. When the experts are full, one common scheme decides which patches get processed and which are skipped. That scheme, Batch Prioritized Routing, ranks patches by the gate's confidence. Cutting expert capacity at test time turns this ranking into a dial that trades compute for accuracy. Nobody has checked whether that ranking resembles what humans find worth keeping. Visual memorability research supplies an independent, human-measured answer to that question. Memorability is the finding that different people reliably remember the same images and video clips, and it can be predicted.

We propose to use memorability as an outside yardstick for how sensible a router is. The central hypothesis is that the two known ways these models fail both flatten the patch ranking and cut it loose from memorability. The first failure is collapse, where a few experts take all the traffic and the rest go unused. The second is homogenization, where balancing penalties force every expert to become the same generalist. If the hypothesis holds, memorability-alignment is a behavioural diagnostic of routing health that load statistics cannot provide. A light training signal that anchors the ranking to memorability should then make capacity cuts lose less of what matters.

We test this in one regime where capacity cuts are unavoidable: context-aware mobile video. There, the compute and bit budgets shrink with the viewer's situation, battery and bandwidth.

Core Claim

The claim has three parts, each falsifiable. First, in a healthy vision mixture-of-experts, a patch's routing priority correlates with human memorability of the content it covers. Second, collapse and homogenization measurably weaken this correlation, and they do so before they visibly hurt classification accuracy. That makes memorability-alignment an earlier warning of routing damage than accuracy alone. Third, adding a memorability-ranking term to the router lets the model keep more human-relevant content when capacity is cut. The resulting clip-level priority is also a usable, nearly free importance signal for deciding which video segments a streaming client should protect.

If the first part fails, meaning router priority is unrelated to memorability even in well-trained models, the project still yields a clear negative finding about what routers optimize. Parts two and three would then be recast as a pure intervention study.

Where This Sits

Sources of the memorability signal. Memorability is measured by showing people a stream of images or clips and counting how often they correctly flag repeats. This produces the LaMem, MemCat, Memento10k and VideoMem benchmarks. Supervised predictors come close to the ceiling set by agreement between two random halves of the observers. Understanding and Predicting Image Memorability at a Large Scale reports MemNet at Spearman 0.64 against a human ceiling of 0.68 on LaMem. Is Image Memorability Prediction Solved? reports 0.67 with a ResNet feature plus boosted-tree regressor. For video, Multimodal Memorability reaches 0.663 on Memento10k against a human ceiling of 0.730, using a dense three-stream network (frames, optical flow, 3D convolutions). These papers supply our labels and our dense reference predictors. They are not sparse models, and they say nothing about routing.

The closest prior work. Traces of Image Memorability in Vision Encoders trains sparse autoencoders on frozen dense vision transformers such as CLIP. It finds that how badly the autoencoder reconstructs an image predicts memorability: up to 0.57 on MemCat and 0.449 when transferred to LaMem. It also finds that low "patch uniformity" predicts memorability. That result motivates our hypothesis, since a router with flat priorities is exactly a uniform treatment of patches. It also gives us a label-free, per-patch memorability "teacher". The paper stays with dense encoders and never asks whether a sparse model's routing decisions carry the same trace.

Routing and collapse, which supply our models and constrain the design. Scaling Vision with Sparse Mixture of Experts introduces Batch Prioritized Routing and test-time capacity reduction. It shows that dropping low-priority background patches is cheap, which makes it the natural object of study. Routers in Vision Mixture of Experts shows that two alternatives remove load collapse and beat the default "each token picks its experts" routing. In Expert Choice routing, each expert picks its tokens; in Soft MoE, experts see weighted blends of tokens. This constrains us: we cannot claim to fix load collapse, because it is already fixed. Our target is different. We care about whether the ranking of tokens is meaningful, which balanced load does not guarantee.

Advancing Expert Specialization for Better MoE diagnoses homogenization under balancing losses. It counters it with two added penalties: an orthogonality loss on expert outputs and a variance loss on gate scores. It reports up to 23.79% relative gains in language models and names vision as open. We port both losses as competitors. A Recipe for an Elastic Mixture shows that randomly masking experts during training lets one checkpoint serve many budgets: validation loss 1.78 versus 2.75 unadapted at half the experts. We borrow this so that capacity cuts at test time are in-distribution.

The application regime. Enhancing QoE for Mobile Users by Environment-Aware HTTP Adaptive Streaming classifies the viewing context from the phone's accelerometer and microphone. It favours smooth playback over sharpness when the viewer is shaken and distracted. This modification of the BOLA buffer-based rate algorithm raises mean opinion score (MOS) from 3.09 to 3.79 in dynamic, noisy settings and from 3.79 to 4.05 in quiet ones. Like its relatives (Human-Centric Resource Allocation in the Metaverse, Toward Cost-Effective Mobile Video Streaming), it treats all segments as interchangeable bytes. We keep its context logic unchanged and only add a per-segment importance term.

Research Program

Step one: the diagnostic. We train a family of ImageNet-1k vision mixture-of-experts models at ViT-S and ViT-B scale, in the style of the V-MoE paper, with routing health deliberately varied:

  • Standard per-token routing with the usual load-balancing loss.
  • The same model with no balancing, which is induced to collapse.
  • The same model with very strong balancing, which is induced to homogenize.
  • Expert Choice routing.
  • Per-token routing plus the orthogonality and variance losses.

For each model we compute two things per patch. The first is routing priority, defined as the maximum gate weight used by Batch Prioritized Routing, or the selection count under Expert Choice. The second is the teacher's memorability map: per-patch sparse-autoencoder reconstruction loss on CLIP, following the Traces paper. At image level we pool priorities and correlate them with LaMem and MemCat human scores. We also record the usual health measures: entropy of expert load, overlap between experts, and gate variance. The test is whether memorability-alignment moves with health and shifts earlier in training than ImageNet accuracy.

Step two: the intervention. We add a ranking term to the router so that patch priority orders patches the way the teacher does. For patches $i,j$ in an image with teacher memorability $m_i > m_j$ and router priorities $p_i, p_j$, the term is:

$$ \mathcal{L}_{\text{mem}} = \sum_{(i,j):\, m_i > m_j} \log\big(1 + e^{-(p_i - p_j)/\tau}\big), $$

where $\tau$ is a temperature. It is added with weight $\lambda$ to the task and balancing losses. The term shapes only the ordering of patches, not which expert each patch goes to. We still expect it to reduce homogenization, because a decisive, content-dependent ordering is incompatible with flat gates. Checking whether this is true is part of the study. All models are trained with random expert masking so that reduced capacity is in-distribution.

Step three: video. We extend the best configurations to frame patches of short clips and fine-tune a memorability regression head on Memento10k and VideoMem. We then sweep capacity down to 25% of tokens processed. We measure three things at each capacity: Spearman correlation with human memorability, action and scene recognition accuracy, and a new "memorable-content retention" score. Retention is the fraction of teacher-memorable patches that survive the cut.

Step four: streaming. Clip-level pooled priority becomes a segment importance weight in the environment-aware BOLA objective. In quiet contexts, bitrate is protected on high-importance segments. In dynamic contexts, stalls are kept away from them. The system is tested in simulated trace-driven sessions and in a small user study. Participants view in a seated condition and a condition with vibration and noise added, then take a recognition memory test on the segments they watched and give a MOS rating.

Baselines

The naive reference points are three. The first is the full-capacity model, which sets the upper bound for accuracy and memorability. The second is random patch dropping at matched capacity, the "router knows nothing" floor. The third is uniform dropping of every other frame.

The strongest existing competitor for the capacity question is V-MoE with Batch Prioritized Routing and its standard balancing loss, as described in Scaling Vision with Sparse Mixture of Experts. It is the published method built precisely for dropping patches under reduced capacity. The V-MoE paper reports that V-MoE matches dense ViT-H with under 50% of the FLOPs and reaches 90.35% ImageNet top-1 at 15B parameters. We found no reported result on Lacuna for a vision mixture-of-experts on Memento10k or VideoMem, so that comparison is new here.

Close behind are Expert Choice routing, where Routers in Vision Mixture of Experts reports Soft MoE at 50.53% versus per-token routing at 48.86% on JFT-300M with a B/16 backbone, and the orthogonality-plus-variance losses of Advancing Expert Specialization for Better MoE. Every router variant shares one backbone, one data order and the same random expert masking. Each gets the same hyperparameter search budget: equal trials over learning rate and auxiliary-loss weight.

Beating these baselines means two things at 50% and 25% capacity, with non-overlapping seed intervals. First, higher Spearman correlation with memorability on Memento10k. Second, higher memorable-content retention, with recognition accuracy no worse than 0.5 points below the best competitor.

For the memorability prediction itself, the dense reference is SemanticMemNet at 0.663 on Memento10k, from Multimodal Memorability. We do not expect to beat it. We report how close a sparse model at a fraction of the compute gets.

For streaming, the baseline is the environment-aware BOLA from Enhancing QoE for Mobile Users, with its reported MOS of 3.79 in dynamic, noisy contexts and 4.05 in quiet ones. It is re-run on our traces and our participants. Beating it means higher recognition accuracy on the watched segments at equal or better MOS and equal or lower stall count.

The key ablation runs the full pipeline with the ranking term replaced by one that matches a shuffled teacher map. This keeps the regularizing pressure of an added loss but removes the memorability content. A second ablation substitutes a low-level saliency map as teacher, to separate memorability from simple contrast.

Why not just…?

Why not just use Expert Choice or Soft MoE, which already solve collapse? They solve load imbalance by construction, as Routers in Vision Mixture of Experts shows. But balanced load says nothing about whether the tokens an expert chooses are the right ones to keep when capacity shrinks. Expert Choice is therefore included as a model in the step-one diagnostic, not just as a baseline to beat. If its selections already align with memorability as well as our ranking term does, that is a real and publishable outcome. The intervention would then be unnecessary for that router, and the diagnostic contribution would stand alone.

Why not just run a separate dense memorability predictor and drop tokens by its map? This is the most serious threat. A linear probe or sparse autoencoder on CLIP (Traces of Image Memorability) or a MemNet-style regressor could supply patch importance directly. We run exactly this "external oracle" condition at matched total FLOPs, counting the oracle's own cost. The external predictor may still win on quality. In that case our claim shrinks to efficiency: the anchored router provides the ranking without a second network, which matters only where the second network's cost is binding. We will report the FLOPs at which the two cross over rather than assert that they do.

Why not just ship several small dense models, one per context budget? For a fixed set of device tiers this is a legitimate design. We compare against dense ViTs at each matched FLOP level. The evidence against it concerns runtime switching rather than quality. A Recipe for an Elastic Mixture shows that a single checkpoint adapted with masking degrades gracefully, whereas an unadapted one does not. A viewer moving between a quiet room and a bus changes budget within a session, and swapping checkpoints mid-stream has a cost we measure directly.

Why not just use saliency to decide what to protect? The memorability literature separates the two: images can grab attention and still be forgotten, as argued in Understanding and Predicting Image Memorability at a Large Scale and What Makes an Object Memorable?. The saliency-teacher ablation tests whether that separation matters for routing. If a saliency teacher matches the memorability teacher, the human-memory framing adds nothing, and we will say so.

Decisive Checks

The first check is correlational and cheap: across the step-one model family, does pooled router priority correlate with LaMem and MemCat scores, and does the correlation fall in the collapsed and homogenized models? A flat result across healthy models rules out the diagnostic reading of the project and leaves only the intervention.

The second check is whether memorability-alignment degrades before ImageNet accuracy during induced homogenization. This is the property that would make it useful as a training monitor.

The third check is the capacity sweep on Memento10k and VideoMem against Batch Prioritized Routing, Expert Choice, the specialization losses, the external-oracle condition and the shuffled-teacher ablation.

The fourth check tests whether the signal matters to people. In the streaming user study, protecting high-priority segments has to raise recognition memory for those segments without lowering MOS in either context. The dynamic, noisy condition is the most informative. There the context-aware baseline already sacrifices sharpness, and the open question is whether keeping stalls away from memorable moments helps memory beyond what smooth playback already gives.

VideoMem's gap between short-term and long-term memory, with a correlation of only 0.305 between them (VideoMem), cautions that any effect may be specific to short-term recall. We therefore report both horizons where the labels allow it.