Convolutional Stages as the Precondition for Self-Discovered Part Masking

Masked autoencoders learn visual features by hiding most of an image and training a network to fill it back in. Two separate fixes have been proposed for their weaknesses. One changes what is hidden: instead of hiding random squares, hide whole object parts such as a bird's head or an entire foreground object, so the network cannot cheat by blurring neighbouring pixels into the gap. The other changes the network: put convolutional layers, which build in the assumption that nearby pixels belong together, in front of the transformer, so it does not have to learn that from scratch. This proposal argues that the two fixes are not interchangeable. The convolutional layers are what make the first fix work when data is scarce. Methods that find parts with no outside help, by grouping the network's own patch features, depend on those features already clustering into coherent regions. A plain vision transformer trained on a few thousand images may never reach that point. A hybrid convolution-transformer should reach it early.

We would build a part-masking variant of the hybrid masked autoencoder ConvMAE. It discovers parts from its own mid-level features, with no pretrained teacher. We would test it in a factorial study that crosses backbone (plain transformer or convolutional hybrid) with masking (random, contiguous-random, or part-based) across pretraining set sizes from full ImageNet down to a 20,000-image mammography corpus. The central prediction is a statistical interaction. Part masking should help the hybrid by roughly the same amount at every data scale, while its benefit on the plain transformer should shrink, and possibly vanish, as data shrinks. If this holds, it settles an open question in the literature: whether semantic masking can stand in for architectural priors in low-data settings, or whether it only pays off once those priors exist.

What is known and what is missing

Part-aware masking has been shown to beat random masking on plain vision transformers, the attention-only image models that treat an image as a flat sequence of patches. SemMAE learns about six parts per image using a frozen iBOT teacher and a StyleGAN decoder, then masks whole parts late in training. It reports 84.5% ImageNet fine-tuning accuracy against 83.1% for MAE. Evolved Part Masking derives parts from the transformer's own attention through an iterative graph cut and reports 84.1% at a training-time cost of about 12%. Self-Guided MAE observes that MAE's patch embeddings cluster on their own within about 50 epochs. It detects this with a decoder "exploitation rate", then switches to masking a foreground cluster found by Normalized Cut, a spectral method that splits a similarity graph into groups. This costs roughly 0.25% overhead and gives 83.2% against 82.5%. All three supply our method, and all three are confined to plain transformers trained on ImageNet-scale data. Self-Guided MAE also reports that its two-way foreground/background split falls back to random-like behaviour on cluttered scenes.

On the architecture side, ConvMAE and its conference version MCMAE use two convolutional stages followed by transformer blocks. They use masked convolutions that only read visible pixels, and they draw the random mask at the coarsest stage and upsample it to the finer stages. They report 85.0% ImageNet fine-tuning, 53.2 box AP on COCO after 25 epochs (MAE needs 100 epochs to reach 50.3), and 51.7 mIoU on ADE20K. Their ablation is the key constraint on our design: replacing masked convolutions with ordinary convolutions drops accuracy from 84.6% to 81.5%, because visible pixels leak into masked regions. Their masks are entirely random.

The motivating negative result comes from Benchmarking Transfer Learning for Dense Breast Tissue Segmentation on Small Mammogram Datasets. With 596 labelled and 20,000 unlabelled mammograms, generic masked image modelling gave negligible or negative gains for transformers, and convolutional networks won. On Data Scaling in Masked Image Modeling similarly finds that masked pretraining overfits on small ImageNet subsets: SwinV2-L reaches 41.9 ADE20K mIoU with 10% of the data and 51.2 with ImageNet-22K. DeiT documents the general data hunger of transformers without spatial priors.

Nobody has combined part masking with a convolutional hybrid. More importantly, nobody has asked whether part masking's benefit depends on the backbone's built-in priors when data is limited. We do not claim the combination itself as the contribution, since it is the obvious next step. The contribution is the interaction hypothesis, plus the measurements that would confirm or refute it.

Core Claim

Self-discovered part masking needs patch features that already group into coherent regions. Convolutional stages produce such features after far fewer images and epochs than global attention does. The theoretical result of Cordonnier et al., that attention can express convolution but is slower to optimise into it, gives a reason to expect this. Two consequences follow. In a hybrid, the switch from random to part masking can happen earlier and will produce better parts. In a plain transformer trained on small data, the switch either never triggers or triggers on noise-level clusters, so part masking behaves like random masking. The accuracy gain from part masking should therefore be roughly constant for the hybrid and should decay with data size for the plain transformer.

Research Program

The method, which we call Part-ConvMAE, keeps ConvMAE's architecture and masked convolutions unchanged. It differs only in how the stage-three mask is chosen.

Masks are defined on the 14×14 grid of stage-three tokens and upsampled to stages one and two exactly as in ConvMAE. Any union of stage-three tokens, however irregular its outline, is a legal mask for masked convolution. The leakage that ConvMAE warns about is therefore avoided by construction, and part boundaries are kept at token resolution. During a short random-masking warm-up, the model computes a part-emergence signal from the stage-three embeddings. We use both Self-Guided MAE's exploitation rate and a cheaper measure, the stability of the spectral clustering across augmented views of the same image. Once the signal crosses a threshold, each image's stage-three token-similarity graph is split into $k$ groups by spectral clustering, with $k$ chosen per image from the eigengap, the largest jump between consecutive eigenvalues of the graph. Whole groups are then masked in random order until the target ratio of 75% is met, and the last group is filled only partially. Using a per-image $k$ instead of a two-way split is meant to address the clutter failure of Self-Guided MAE. Both the warm-up and the clustering use the model's own features, so no teacher network is needed.

The analysis has three parts. First, we measure when part emergence occurs, in epochs and in images seen, for ViT-B versus ConvMAE-Base at each data scale. Second, we measure part quality, scored zero-shot as the agreement between discovered clusters and CUB part annotations and ImageNet foreground boxes. Third, we measure downstream accuracy. The hypothesis predicts that emergence time and part quality explain most of the gap in downstream gains between the two backbones.

Baselines

The naive reference point is supervised training from ImageNet initialisation with no in-domain self-supervision. On mammograms this is the setting the benchmark paper found hard to beat with transformers. Every method must at least improve on it there.

The strongest existing method on the architectural side is ConvMAE-Base with random block masking, rerun in our code. The rerun uses the same data subsets, the same number of epochs (400 on ImageNet scales and a matched budget on mammograms), the same mask ratio, and the same fine-tuning recipe, with an equal hyperparameter sweep for every arm. The reported reference is 85.0% ImageNet fine-tuning accuracy (ConvMAE), together with 51.7 mIoU on ADE20K and 53.2 box AP on COCO. Beating it on full ImageNet means a fine-tuning gain of at least 0.5 points over our own rerun, averaged over three seeds and outside seed variance. The more important targets are the gains on 10% ImageNet and on mammograms.

The strongest part-masking competitor is Self-Guided MAE on ViT-B. It needs no teacher and adds little overhead, it reports 83.2% fine-tuning, 45.2 ADE20K mIoU, and 43.3 COCO box AP, and we rerun it under the identical budget. We also report SemMAE's 84.5% as a teacher-assisted reference, and in the low-data arms we run a version of it that uses parts from a frozen DINO-style teacher.

For the mammography task, the strongest reported system is EfficientNet with multi-view self-supervision and full fine-tuning, which reaches a Dice score of 0.837 and reduces density mean absolute error from 14.81% to 11.78% (Benchmarking Mammograms). Matching 0.837 with a hybrid masked autoencoder would already reverse that paper's negative finding for masked pretraining; exceeding it would be a clear win.

The ablation that isolates the key ingredient is size-matched contiguous random masking. It masks randomly placed blobs whose number and sizes match the discovered parts of each image but ignore image content. A second ablation keeps everything else fixed but uses a two-way Normalized Cut in place of the adaptive-$k$ clustering.

Why not just…?

Why not just raise the random mask ratio to 85–90%? Kong et al. argue that high ratios already remove local shortcuts. However, SemMAE and Self-Guided MAE each beat random MAE at matched ratios. We include ConvMAE at 85% and 90% as extra arms. If high-ratio random masking matches Part-ConvMAE at small data scales, the interaction claim fails for the hybrid, and we would report that outcome.

Why not just mask large contiguous blocks? This is the most serious threat, because part masking may help simply by removing contiguous regions so that neighbouring visible pixels cannot be interpolated across the gap. The size-matched contiguous ablation settles it directly. If that ablation captures most of the gain, the conclusion becomes that convolutional backbones need shape-aware random masking, not semantics, and we would report it that way.

Why not just use a convolutional network on small medical data? The benchmark paper shows that convolutional networks win there, and EfficientNet is our reference. The question is whether masked pretraining can add value once the right prior is present. If Part-ConvMAE does not exceed 0.837 Dice, the practical recommendation stays with plain convolutional networks.

Why not just take parts from a pretrained teacher? Teacher-supplied parts may work as well as or better than self-discovered ones, and we run them. A teacher matching our method would not contradict the claim about priors. It would show that the bottleneck is part quality, which the backbone supplies for free when no teacher exists, as in many medical domains.

Proposed Experiments

We pretrain ViT-B and ConvMAE-Base under random, contiguous-random, self-discovered part, and teacher part masking. Pretraining sets are 10%, 25%, and 100% of ImageNet-1K and the 20,000 unlabelled mammograms. We evaluate by ImageNet fine-tuning and linear probing, CUB-200 fine-tuning, ADE20K mIoU, COCO box and mask AP on the full-data models, and mammogram Dice and density error.

The decisive check is the backbone × masking interaction on accuracy. A part-masking gain on the hybrid that holds steady from 100% down to 10% of ImageNet and on mammograms, together with a ViT gain that falls toward zero, would confirm the claim. Parallel gains on both backbones would refute it. A second decisive check is mechanistic: emergence epoch and part-annotation agreement should be markedly better for the hybrid at small scales and should predict the downstream gain across all runs. A third check covers clutter: on COCO-style multi-object images, adaptive-$k$ clustering should beat two-way Normalized Cut, or the per-image part count adds nothing.