Many large vision models are built as mixtures of experts. An image is cut into small patches, and a small learned module, the router, sends each patch to one or a few of many sub-networks (the experts). To stop the router from sending everything to a few favourites, a problem called expert collapse, training normally adds a load-balancing penalty that pushes patches to spread evenly across experts. In photographs of landmarks, people or vehicles, though, most patches are sky, road or foliage. We think the even-spreading rule then forces every expert to spend its capacity on near-identical background patches, which makes the experts alike. The same background patches also pollute the image descriptor used for search. In retrieval, two unrelated scenes look similar because their skies match, and two photos of the same building look different because their surroundings differ.
This note proposes one change that targets both problems. We add a zero-compute "sink" expert that is exempt from the balancing penalty. The router learns to send low-information patches there. The real experts are balanced only over the patches that remain. As a by-product, the fraction of each patch sent to the sink across layers gives a foreground mask with no box supervision, and the retrieval descriptor pools only the non-sink patches. The claim to test is specific. Standard balancing gets worse at keeping experts distinct as the background fraction of images grows. Exempting a sink removes that dependence. The resulting mask cleans descriptors at least as well as attention-based masking in dense models.
Core Claim
Prior work treats expert collapse as a problem of optimization dynamics. It arises from the rich-get-richer feedback noted in Scaling Vision with Sparse Mixture of Experts, or from class entanglement, which Mixture of Experts with Soft Nearest Neighbor Loss addresses by disentangling features before routing. The fixes proposed to date share one assumption: every patch deserves real expert compute. Examples include the orthogonality and variance losses of Advancing Expert Specialization for Better MoE and the entropy-barrier balancing of φ-Balancing for Mixture-of-Experts Training. Those two papers motivate our concern, because both show that forcing uniform allocation produces experts with overlapping, near-collinear outputs.
Our hypothesis is that in cluttered imagery this pressure is worse than in text. A majority of patches are homogeneous background, and balancing distributes them across all experts. The retrieval literature supplies the second half of the argument. Interferences in Match Kernels. and Object Level Deep Feature Pooling for Compact Image Representation show that background regions produce false matches when an entire image is pooled. The same patches that we suspect homogenize experts are the ones that pollute the descriptor. One learned decision, whether a patch goes to the sink, should address both.
The closest existing observation comes from V-MoE. Its Batch Prioritized Routing drops low-priority patches, often background, when expert buffers overflow. There, dropping is a side effect of hardware capacity. It happens under a standard balancing loss, and V-MoE is evaluated only on classification. We make the drop an explicit, learned route. We remove it from the balancing penalty, reuse it as a pooling mask, and test it on de-duplicated instance retrieval.
Research Program
The architecture is a ViT-B/16 in which every other block has an MLP mixture of experts with $E$ experts and top-1 routing, plus one sink route. The sink is the identity, so a sink-routed patch passes only through the residual connection. For a batch of patches, let $s(x)$ be the router probability that patch $x$ goes to the sink. Let $T$ be the set of patches not routed to the sink, $f_e$ the fraction of $T$ dispatched to expert $e$, and $P_e$ the mean router probability of expert $e$ over $T$. The balancing loss is
$$L_{bal} = E \sum_{e=1}^{E} f_e P_e + \lambda \max(0, \bar{s} - \rho),$$
where $\bar{s}$ is the mean sink probability over the batch. The first term is the usual Switch-style balance, restricted to $T$. The second term is a soft cap that keeps the sink from absorbing everything once its share exceeds a budget $\rho$, and $\lambda$ weights it. We sweep $\rho$ rather than tune it per dataset. If retrieval quality depends sharply on $\rho$, that counts as evidence against the method.
For retrieval, each patch gets a keep-weight $w(x) = 1 - \bar{s}_L(x)$, where $\bar{s}_L(x)$ is its sink probability averaged over the last few MoE layers. The descriptor is generalized-mean pooling of final patch features weighted by $w$.
We pretrain the MoE and all dense baselines on ImageNet-21k with matched compute. We then fine-tune on RGLDv2-clean, the Google Landmarks v2 split that On Train-Test Class Overlap and Detection for Image Retrieval purged of the 18 landmark classes that leak into Oxford and Paris. We use that paper's image-level training recipe so that only the backbone and pooling differ.
The analysis side matters as much as the build. We measure expert homogenization with pairwise cosine similarity between different experts' outputs on the same patch, following the overlap measure of Advancing Expert Specialization for Better MoE. We also track routing entropy and the Gini coefficient as in φ-Balancing for Mixture-of-Experts Training. Foreground ground truth comes from the manual boxes in Google Landmark Boxes (Detect-To-Retrieve), which we use only for evaluation and never for training.
Baselines
The naive reference point is a dense ViT-B/16 with plain generalized-mean pooling over all patches, trained with the same recipe. This measures how much clutter costs a whole-image descriptor. For scale, the file reports ResNet101 with GeM at 64.7 mAP (Medium) and 38.5 mAP (Hard) on ROxf in Revisiting Oxford and Paris. That model was trained on different data, so it is context rather than a matched comparison.
The strongest existing method is CiDeR from On Train-Test Class Overlap and Detection for Image Retrieval. It is a dense single-stage network that thresholds its own spatial attention maps at several levels into soft masks, trained from image labels alone. It reports 61.9% mAP on ROxf-Hard when fine-tuned on RGLDv2-clean. It is the right rival because it solves the same problem, removing clutter without boxes, and does so in a cheap dense model. We will reimplement its attentional localization on our dense ViT-B/16 with the same pretraining, data, epochs and hyperparameter search budget. Beating it means exceeding its ROxf and RPar Hard mAP under that matched setting, and also against its published 61.9 if our reproduction falls short.
The standard vision MoE baseline is the same MoE with top-1 token choice, a full-batch balancing loss and Batch Prioritized Routing, as in V-MoE. It is pooled over all surviving patches. It is run at the same expert count, capacity and compute.
Two further MoE baselines address collapse directly. The first is Expert Choice routing (Mixture-of-Experts with Expert Choice Routing; Routers in Vision Mixture of Experts). Each expert picks its top patches, so no balancing loss is needed and a patch can be picked by no expert. The second adds the orthogonality and variance losses of Advancing Expert Specialization for Better MoE to standard token choice. Both get the same tuning budget as our method.
Three ablations isolate the key ingredient. One keeps the sink but counts it inside the balancing loss, which removes the exemption. A second keeps the exemption but pools over all patches, which removes the mask. A third derives a mask from a standard MoE's low maximum gate value, which asks whether any router confidence would do. As a secondary transfer check, we compare against EDITOR (Magic Tokens), which reports 65.7% mAP on RGBNT201.
Why not just…?
Why not just mask background with attention in a dense model, as CiDeR does? This is the real threat. If a dense ViT with CiDeR-style masking matches our retrieval mAP at equal compute, then the retrieval half of our claim fails. The collapse finding would survive only as a statement about how to train vision MoEs. We do not argue around this. The matched CiDeR reimplementation is the headline comparison. Our structural reason for expecting an edge is modest. V-MoE reports that experts specialize by depth, with early layers on texture and late layers on semantics. A per-layer sink can keep background as context early and drop it late. A single attention cut cannot. The per-layer sink-rate curves will show whether this actually happens.
Why not just use Expert Choice, which already lets background go unselected? Expert Choice fixes each expert's capacity. With roughly 80% background, the experts must fill their quotas with background whenever the capacity factor exceeds the foreground share. The model therefore cannot adapt its drop rate to each image. We test this by sweeping capacity factor and comparing homogenization and mAP against our method. If Expert Choice at its best capacity matches us, the exemption adds nothing, and we will report that.
Why not just add orthogonality or entropy regularizers to fight homogenization? These losses spread experts apart. They still require experts to process background, and they do not produce a mask. If they reduce expert similarity as much as the sink does, our explanation of collapse is wrong. In that case the sink's value reduces to its mask, which the ablation that pools over all patches separates.
Why not just crop with a landmark detector? Detect-to-Retrieve needs 94,000 box-annotated images and a second network at indexing time, and failed proposals cannot be recovered. The file's evidence, CiDeR at 61.9 against a previous best of 49.6 on ROxf-Hard, already favours single-stage masking. We therefore use the boxes only to score masks, not as a competing pipeline.
Decisive Experiments
The first experiment is a controlled clutter sweep. We paste ImageNet objects onto scene backgrounds so that the foreground covers 80%, 50%, 30% or 20% of the image. We then train standard-balanced MoE, Expert Choice, orthogonality-regularized MoE and sink-exempt MoE at matched compute. The prediction is that expert-output similarity under standard balancing rises as foreground shrinks, while under the sink it stays roughly flat. If standard balancing shows no such trend, the collapse hypothesis is false and the project narrows to retrieval masking.
The second experiment checks the mask. On Google Landmark Boxes images, we compute the overlap between sink-routed patches and patches outside the boxes, layer by layer. We compare this against CiDeR's attention masks and the low-gate mask.
The third experiment is retrieval. We report mAP on ROxf and RPar, Medium and Hard, with and without the R1M distractors, after RGLDv2-clean fine-tuning, alongside FLOPs and measured wall-clock time. Wall-clock matters because V-MoE and Spatial MoE both note that FLOP savings do not always become speed. Success means beating matched CiDeR on the Hard protocols and against R1M, with the no-exemption ablation falling back toward the standard MoE. If only the mask helps and the exemption does not, we will report that the collapse half of the story was wrong.
References
- Scaling Vision with Sparse Mixture of Experts
- Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
- Advancing Expert Specialization for Better MoE
- $\phi$-Balancing for Mixture-of-Experts Training
- Interferences in Match Kernels
- Object level deep feature pooling for compact image representation
- On Train-Test Class Overlap and Detection for Image Retrieval
- Detect-To-Retrieve: Efficient Regional Aggregation for Image Search
- Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking
- Mixture-of-Experts with Expert Choice Routing
- Routers in Vision Mixture of Experts: An Empirical Study
- Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification