Weight-Shared Encoder Expansion for Masked Pre-training of Tiny Vision Transformers

Masked image modeling teaches a vision model without labels. Most of each image's patches are hidden, and the model has to reconstruct them. This works well for large Vision Transformers (ViTs, models that treat an image as a sequence of patch tokens), but there is good reason to expect it to fail for tiny ones. Hiding 75% of an image is a very strong regularizer. A separate body of work shows that regularizers sized for big models make small models underfit: they cannot even learn the training distribution. We propose a fix taken from the tiny-network literature. During pre-training only, the small encoder becomes a weight-shared slice of a wider encoder. Both the slice and the wide model reconstruct the masked patches. At test time the wide parts are thrown away, so the deployed model costs exactly what the tiny model costs.

The bet is specific. Masked autoencoders already carry discardable training-time capacity in the form of their decoder, so the claim cannot simply be "add capacity." The claim is about where the extra capacity goes. Capacity placed downstream of the encoder can take over the reconstruction work and leave the encoder's features poor. Capacity placed inside the encoder through shared weights forces the parameters we keep to carry the representation. If that is true, a tiny ViT trained this way should tolerate MAE's standard high masking ratio. It should also beat the best result from the obvious knobs, which are lowering the mask ratio or enlarging the decoder, without needing a teacher model.

Where this sits

The capacity-mismatch literature motivates the proposal and also supplies its method. Revisiting Data Augmentation in Model Compression shows that the best augmentation strength grows with parameter count: RandAugment magnitude 14 helps an unpruned ResNet20 but costs about 2% once the network is heavily pruned. FBNetV3 finds the same capacity–recipe correlation by searching over recipes. Dropout Reduces Underfitting shows that small ViTs gain from turning dropout on only early in training (+0.4% on ViT-T). It explicitly leaves self-supervised learning as an open question. The method itself comes from Network Augmentation for Tiny Deep Learning, called NetAug. NetAug embeds a tiny CNN inside a wider weight-shared network during training and reports +2.2% ImageNet accuracy for MobileNetV3 w0.35, with no inference cost. NetAug also hurts ResNet50 by 0.3%, which is the kind of capacity dependence we will test for. NetBooster reaches a similar result through expand-then-fold layers. Both papers are supervised and use CNNs. Neither touches ViTs or reconstruction objectives.

On the masked modeling side, Masked Autoencoders Are Scalable Vision Learners, known as MAE, sets the design we build on. The encoder sees only the visible 25% of patches, and a light decoder regresses the pixels of the hidden ones. MAE's evidence comes from Base, Large and Huge models. BEiT and MaskDistill swap the target for discrete tokens or teacher features. Both depend on external pre-trained networks, which we avoid.

The closest work is Asymmetric Masked Distillation. It observes that small models struggle at high masking ratios and fixes this with a large pre-trained teacher that sees more of the image than the student. Its "small" model is ViT-Base, and it needs that teacher. We target sub-10M-parameter ViTs, use no teacher, and supply the missing capacity through the student's own widened weights rather than through a teacher's features. Effective Self-supervised Pre-training on Low-compute Networks without Distillation, called SSLight, is our main motivating precedent. It showed that small networks' self-supervised gap in contrastive learning came from a miscalibrated view-sampling regularizer rather than a hard capacity limit, and closed much of the gap without distillation. Its analysis covers multi-crop contrastive and clustering methods only. Masked reconstruction is untouched there.

Core Claim

For tiny ViTs, MAE underfitting comes from a mismatch between how hard the masked task is and how much capacity the encoder has. We claim that adding temporary capacity inside the encoder through weight sharing fixes this mismatch better than the other two ways of fixing it. One alternative is to make the task easier by lowering the mask ratio. The other is to add temporary capacity outside the encoder by enlarging the decoder. We also claim the benefit is capacity-dependent: it should shrink to zero or turn negative at ViT-Base, mirroring NetAug's ResNet50 result.

Research Program

The training objective has one term for the tiny encoder and one for a wider encoder that contains it. Let $\theta$ be the tiny encoder's weights. Let $\phi$ be the extra weights that widen it, meaning additional attention heads, embedding channels and MLP hidden units, so that the wide encoder uses $(\theta, \phi)$. Let $D$ be a shared decoder. For a masked image $x$ with mask ratio $r$, we minimize

$$\mathcal{L} = \ell_r\big(D(E_\theta(x))\big) + \lambda\, \ell_r\big(D(E_{\theta,\phi_s}(x))\big),$$

where $\ell_r$ is MAE's per-patch normalized-pixel mean-squared error on the masked patches. $\phi_s$ is a width multiplier sampled each step from a small set, as NetAug does. $\lambda$ weights the auxiliary term. Gradients from the wide pass flow into $\theta$, so the tiny model gets the optimization benefit of a wider network's loss surface. Only $E_\theta$ is kept for fine-tuning.

MAE's encoder sees only 25% of tokens, so the wide pass is cheap relative to supervised NetAug. We will measure the overhead directly; NetAug reports 16.7% extra training time on CNNs.

The program has three stages. First, a diagnosis stage measures underfitting directly. We record final training reconstruction loss and downstream accuracy for ViT-Tiny, ViT-Small and ViT-Base, across mask ratios from 40% to 85%, with standard MAE. This tests whether the best mask ratio really falls with capacity, which is the premise of the whole proposal and is currently unmeasured. Second, we apply shared encoder expansion and test whether it moves the tiny model's best mask ratio back toward 75%. Third, we test whether a capacity-matched rule for the expansion width and $\lambda$ transfers across sizes without re-tuning. Hyperparameter brittleness is the main weakness the file identifies in SSLight and NetAug, so this transfer test matters.

Baselines

The naive reference point is MAE run on ViT-Tiny exactly as published, with a 75% mask, the standard decoder and the standard recipe. It shows what happens with no adjustment.

The strongest teacher-free competitor is a tuned MAE for ViT-Tiny. Its mask ratio, decoder depth and width, and drop-path rate are searched jointly, using the same search budget we spend tuning $\lambda$ and the expansion widths. This baseline embodies everything a skeptic would try first, so beating it is the real test. We found no reported result on Lacuna for MAE ViT-Tiny ImageNet-1K fine-tuning, so we will establish this number ourselves. For calibration, the nearest published figures are 68.3% ImageNet top-1 for SSLight on MobileNetV2 and 66.8% for SSLight on ResNet18 Effective Self-supervised Pre-training on Low-compute Networks. Supervised ViT-S/16 reaches 80.0% in 300 epochs under the simplified recipe of Better plain ViT baselines for ImageNet-1k. We will rerun SSLight's DINO-based recipe on ViT-Tiny, which that paper also studied, as the strongest non-masked self-supervised competitor.

A teacher-based reference, AMD run with a ViT-Base MAE teacher and a ViT-Tiny student, gives an upper reference that uses extra resources. For context, AMD reports 84.6% ImageNet-1K top-1 when the student is ViT-Base. We would count matching AMD within a point without any teacher as a strong result, not a requirement.

Two ablations isolate the key ingredient. The first is decoder-only expansion: we add the same number of extra parameters and the same FLOPs to the decoder instead of the encoder. The second is an unshared auxiliary encoder: a wide encoder of equal size is trained alongside the tiny one, but without sharing weights with it. If either matches shared expansion, the weight-sharing story is wrong.

For fair comparison, every method gets the same ImageNet-1K pre-training images and data augmentation. Every method also gets equal GPU-hours, not equal epochs: baselines are trained longer to absorb our overhead. Every method then goes through the same fine-tuning protocol. Beating the tuned baseline means at least a 1-point ImageNet fine-tuning gain for ViT-Tiny, averaged over three seeds with non-overlapping seed spread. It also means gains in the same direction on ADE20K mIoU or COCO detection AP.

Why not just…?

Why not just lower the masking ratio for small ViTs? This is the most serious threat. MAE and BEiT argue that low ratios let the model fill in pixels from neighboring patches without learning about objects. AMD notes that keeping the student's task hard mattered in its setting. Neither argument has been tested on ViT-Tiny, so the mask-ratio sweep in our diagnosis stage settles the question directly. If the best low-ratio MAE run matches shared expansion at 75%, we will report that the simple knob suffices and drop the method claim. The diagnosis result would still stand.

Why not just make the decoder bigger? The decoder is already discardable capacity, so this is the natural cheaper alternative. Our hypothesis is that a stronger decoder takes over reconstruction work and leaves the encoder's features worse. That is consistent with MAE's own finding that decoder design affects linear probe quality, but nothing in the file tests it for tiny encoders. The parameter-matched decoder-only ablation measures it head-on. Linear probing is the sharper metric here, because it shows how much the encoder alone has learned.

Why not just distill from a large pre-trained model? Distillation works, as AMD shows. It does, however, require a large teacher, and SSLight found that a corrected from-scratch recipe beat teacher-based methods for ResNet18. NetAug also reports that capacity expansion is complementary to distillation. We therefore treat AMD as a reference and additionally test AMD combined with shared expansion, to see whether the gains add.

Why not just remove dropout and drop-path? Removing regularizers helped small translation models in MobileNMT. Removing them does not remove masking, however, and masking is the dominant regularizer in this setting. Early dropout beat no dropout for ViT-T in supervised training. Our tuned baseline already searches drop-path, including zero and early-only schedules, so this alternative is folded into the strongest competitor rather than argued away.

Proposed Experiments

Pre-training runs on ImageNet-1K for ViT-Tiny, ViT-Small and ViT-Base. Evaluation covers ImageNet fine-tuning and linear probing, ADE20K semantic segmentation in mIoU, and COCO detection in box AP.

The decisive experiment is a grid over mask ratio (40%, 60%, 75%, 85%) and capacity placement (none, decoder-only, unshared auxiliary, shared encoder), run at all three model sizes. The hypothesis predicts a specific pattern. Without expansion, ViT-Tiny's best mask ratio should sit below 75%. Shared expansion should move that optimum back up, and its best cell should exceed every baseline cell for ViT-Tiny. The advantage should shrink for ViT-Small and vanish or reverse for ViT-Base. Any other pattern counts against the hypothesis, in particular a flat mask-ratio curve for ViT-Tiny, decoder-only expansion matching shared expansion, or equal gains at ViT-Base. That last case would mean we found a generic training improvement rather than a capacity-mismatch effect.

The secondary check concerns underfitting itself. Shared expansion should lower the tiny encoder's own training reconstruction loss, measured with the auxiliary branch removed, if the mechanism is relief of underfitting rather than extra regularization.