Data-Tracking Prototypes for Masked Image Modeling

Many self-supervised vision models learn by hiding most of an image and asking a network to predict what was hidden. In one strong family of these methods, masked image modeling with an online tokenizer, the hidden patch is not described by its pixels. It is described by which of several thousand learned "prototypes" it resembles most. A prototype here is a reference vector, something like an entry in a visual vocabulary. Recent work shows that this vocabulary quietly falls apart during training. In iBOT, a model with 8,192 prototypes ends up using only about 1,000 distinct ones, and the rest become duplicates (On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods). The published fix is a loss that pushes prototype vectors apart. It restores uniqueness, but it buys fine-grained in-domain gains at the cost of a slight drop when features are transferred to unrelated domains.

We propose that how diversity is obtained matters as much as how much of it there is. A loss that pushes prototypes apart spreads them evenly over the space, including regions with little data. An alternative keeps prototypes out of the gradient entirely and moves each one toward the patches it is actually responsible for. Prototypes updated this way are diverse but follow where the data lies. We will build the second option for the patch-level tokenizer of masked image modeling, using an online mixture-model update on the unit sphere. We will then test a specific hypothesis: diversity that tracks the data gives the long-tail gains of repulsion without the transfer penalty, and diversity alone (random, fixed prototypes) does not.

Core Claim

The partial collapse of patch prototypes in iBOT-style masked modeling is caused by training the encoder and the prototypes jointly through the same gradient. The best remedy is therefore to change how prototypes are updated, not to add a repulsive loss. Concretely, we claim that replacing gradient updates of iBOT's patch prototypes with a decoupled spherical online EM update (EM: expectation-maximization) will:

  1. Keep nearly all prototypes unique.
  2. Match or exceed KoLeo-proto, the repulsion loss of On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods, on long-tailed fine-grained recognition.
  3. Avoid KoLeo-proto's out-of-domain degradation on dense and unrelated-domain transfer.

The third point is the new scientific content. It turns the specialization-versus-generalization trade-off reported in that paper into a testable question about the mechanism of diversity.

Relation to Prior Work

Three papers motivate the proposal, and one supplies its method.

Motivating evidence. KoLeo-proto identified partial collapse in iBOT and fixed it with a nearest-neighbor entropy loss on the prototypes. We keep its diagnosis but replace its remedy, and we treat its transfer drop as the phenomenon to explain. Img2Vec showed that masked-modeling students learn better when their targets are diverse across patches, though it got that diversity from an external ConvNet teacher. We instead make the model's own targets diverse. MOCA predicts masked-patch assignments to a codebook made of a first-in-first-out queue of recent teacher features. That queue follows the data, but only passively: it mirrors whatever is frequent, so background-heavy patch statistics should dominate it.

Supplied method. Why Prototypes Collapse traced partial collapse to joint encoder–prototype gradients. Its fix, called Full Decoupling, updates prototypes as Gaussian mixture components by online EM with forgetting. It reached 100% uniqueness on the CARP method and added 3.0 points on iNaturalist 2018. Its authors evaluated only global, image-level representations, left dense patch-level models as future work, and flagged spherical (von Mises–Fisher) distributions as unexplored. We do both.

Contrasting design. Self-Organizing Visual Prototypes abandons learned prototypes for a 65,536-entry memory bank, including a masked-modeling variant. We keep a compact, parametric vocabulary and ask whether its failure is fixable.

Research Program

The base system is iBOT (Image BERT Pre-training with Online Tokenizer). In iBOT, a slowly updated (exponential moving average) teacher assigns each patch a distribution over $K$ prototypes, and the student predicts that distribution for masked patches. We change only how the patch prototypes are updated.

Prototypes $\mu_k$ live on the unit sphere and receive no gradient. At each step we draw a random subset of unmasked teacher patch embeddings $z_n$. This keeps cost manageable, because a batch contains about 196 patches per image. We compute responsibilities $r_{nk} = \mathrm{softmax}_k(\kappa\, \mu_k^\top z_n)$, where $\kappa$ is a shared concentration. This is the posterior under a von Mises–Fisher mixture, and it coincides with iBOT's temperature-scaled softmax. We then update running sufficient statistics with forgetting factor $\lambda$:

$$ s_k \leftarrow \lambda s_k + \sum_n r_{nk} z_n,\qquad \mu_k = s_k/\|s_k\|. $$

The responsibility mass $m_k \leftarrow \lambda m_k + \sum_n r_{nk}$ is tracked so that prototypes whose mass decays can be re-seeded from poorly explained patches. This follows the responsibility-based forgetting of Full Decoupling, transferred to the sphere.

During each encoder step the prototypes are fixed. They move only through this statistics update, never through backpropagation.

Separating patch and global heads. Because the global head and the patch head can collapse differently, we first measure uniqueness in each one separately for vanilla iBOT. Uniqueness is the fraction of prototypes with no duplicate within a cosine threshold. We also measure effective vocabulary size, the exponential of the entropy of average assignments. The file does not say whether the reported ~1,000-of-8,192 figure refers to one head or both, so this measurement is itself a deliverable.

Measuring where prototypes go. We will measure what the prototypes cover using ImageNet-9's foreground/background mixing splits. For each prototype, we compute the fraction of its assignment mass that falls on foreground patches. Our prediction is specific. Vanilla iBOT and the FIFO codebook will spend most prototypes on background texture. KoLeo-proto will spread prototypes but leave many with near-zero mass. Decoupled EM will give more prototypes nonzero foreground mass.

Baselines

Unmodified iBOT (the do-nothing reference). We run iBOT with the same 8,192-prototype patch head, trained with ViT-S on ImageNet-1K under the published recipe and our epoch budget. This tells us whether any intervention helps at all. For context, iBOT reports 79.5% ImageNet linear probe with ViT-B, 50.0 mIoU on ADE20K, and 49.4 box AP on COCO with ViT-S (Image BERT Pre-training with Online Tokenizer). We will compare against our own re-run at matched epochs rather than these figures.

iBOT with KoLeo-proto (the strongest existing competitor). This is the one published remedy for exactly this collapse in exactly this architecture. With ViT-S on iNaturalist-2018 linear probing, it reports 51.4% against 50.1% for iBOT-vMF (On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods). We will apply it to the patch head, the global head, and both. Its weight gets the same tuning budget as our forgetting factor: the same number of short-schedule sweeps, selected on held-out ImageNet k-NN.

Beating it means both of the following:

  1. Equal or better iNaturalist-2018 linear probing.
  2. A statistically separable advantage, across three seeds, on at least two out-of-domain transfer tasks, specifically ADE20K, Cityscapes, and Stanford Cars, where KoLeo-proto falls below vanilla iBOT.

If KoLeo-proto shows no transfer drop in our runs, the trade-off we target is absent at patch level, and we will report that.

Decoupled Gaussian EM, as in Full Decoupling. This is the exact method of Why Prototypes Collapse, ported to the patch head without the spherical change. It isolates whether the von Mises–Fisher formulation matters, or whether decoupling alone does the work.

MOCA-style FIFO queue codebook. This is a data-following codebook that receives no gradient but has no EM structure (MOCA). It isolates whether responsibility-weighted, forgetting-based updates are better than passively storing recent features.

Frozen random prototypes (the key ablation). We sample 8,192 points uniformly on the sphere and never update them. They are 100% unique by construction, but they ignore the data. If this matches our method, the claim that data-tracking matters is false.

All runs share the data, the masking ratio, the augmentations, the epochs, and the evaluation protocols.

Why not just…?

Why not just increase $K$? KoLeo-proto's own analysis shows that iBOT at $K$=8,192 keeps only about 1,000 unique prototypes, so extra slots become duplicates (On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods). Larger vector-quantization codebooks also degrade through shrinkage (Early Quantization Shrinks Codebook). We will still run iBOT at $K$=16,384 as a cheap check. If its effective vocabulary grows linearly with $K$, this alternative is a threat and we will say so.

Why not just keep centering and sharpening? Centering and sharpening make data spread evenly across prototypes. They do not keep the prototype vectors themselves apart, which is why partial collapse coexists with them (On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods). The vanilla baseline measures this directly at patch level.

Why not just use KoLeo-proto, or a memory bank? KoLeo-proto is the real threat. If it shows no transfer drop at patch level, our method's advantage reduces to engineering, such as removing a loss weight. The KoLeo-proto baseline is designed to settle exactly this. A memory bank as in Self-Organizing Visual Prototypes avoids collapse by holding 65,536 entries. We aim for a compact vocabulary and will report memory use next to accuracy rather than claim superiority.

Isn't this just an exponential-moving-average k-means codebook? The update is close to the moving-average codebook updates used in vector quantization. We do not claim the update rule as new. The contribution is two things. First, its use as the target vocabulary of a masked student. Second, the controlled comparison showing whether data-tracking diversity, rather than repulsion or randomness, resolves the transfer trade-off.

Proposed Experiments

We pretrain ViT-S/16 on ImageNet-1K for 300 epochs with all six variants and three seeds. The two leading variants are then repeated with ViT-B to check that conclusions hold at that scale.

In-domain evaluations. These are ImageNet-1K k-NN and linear probing, and iNaturalist-2018 linear probing, the long-tail test where KoLeo-proto has a reported number.

Transfer evaluations. These are ADE20K semantic segmentation, COCO detection, Cityscapes low-shot segmentation with 100 images, and Stanford Cars and CUB-200 linear probing. For Cars and Cityscapes under these specific SSL models, no reported result was found on Lacuna, so these baselines will be produced by our own runs.

Mechanism diagnostics. We report patch-head uniqueness, effective vocabulary over training, and ImageNet-9 foreground mass per prototype.

Decisive outcomes. The hypothesis is supported if decoupled spherical EM matches KoLeo-proto on iNaturalist, beats it on transfer, and clearly beats frozen random prototypes. It is refuted if random prototypes match it, which would mean uniqueness alone suffices. It is also refuted if it inherits KoLeo-proto's transfer drop, which would mean any patch-level diversity over-specializes. A third diagnostic tests the proposed explanation: if foreground coverage does not differ across methods while transfer does, that explanation is wrong even if the method works.