Masked autoencoders learn image features by hiding most of an image and training a network to fill in the missing patches. These models fine-tune very well. Their frozen features, however, perform poorly when a single linear classifier is trained on top, a test called linear probing. This proposal tests one specific cause of that gap. In a standard Vision Transformer, each patch's location is added directly into its feature vector. Reconstructing a hidden patch depends heavily on where it is (sky tends to sit at the top, ground at the bottom). Under regularized training, the "where" signal is strong enough to take over the largest directions of the feature space and push semantic content into weaker ones. The fix is to keep patch position out of the encoder's features altogether. Position would enter attention only through a separate pathway, as in disentangled attention, and absolute location would be given back only to the reconstruction decoder.
The opportunity comes from extending a theoretical tool to a setting it has not yet covered. The unconstrained feature model treats a network's last-layer features as free variables, so that one can solve exactly for the feature geometry that training prefers. It has been shown to be the correct infinite-depth limit of regularized Transformers, but only for classification. We extend it to masked reconstruction. From that extension we derive a prediction that can be falsified: position-predictable variance in the reconstruction targets competes with content for the encoder's limited feature directions. We then test the prediction by changing only how position enters a masked autoencoder.
Core Claim
Train a masked autoencoder under weight decay with a mean-squared-error reconstruction loss. If position is mixed into the encoder's token features, the dominant directions of those features will encode location and location-by-content interactions rather than object identity. Routing position through a separate attention pathway, so that it never enters the token features, frees those directions for semantics. Re-injecting absolute position only in the decoder, as DeBERTa's Enhanced Mask Decoder does for text, preserves reconstruction quality. Three measurable consequences should follow. First, a much smaller fraction of encoder feature variance should be linearly decodable from patch coordinates. Second, variance along downstream class-separating axes should shrink, which is the "directional collapse" quantity. Third, linear-probe accuracy should rise, all at equal pretraining compute.
The claim concerns where position lives in the network, not any particular attention design. We therefore test several mechanisms that keep position out of the features and expect them to behave alike.
A Masked-Reconstruction Feature Model
The supplying result is Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers. It proves that, as depth grows, regularized Transformers reduce to a generalized unconstrained feature model, with distance to collapse shrinking as $O(L^{-1/2})$ in depth $L$. The proof assumes one categorical label per context and states that reconstruction objectives are left open. We adopt the free-feature abstraction for reconstruction targets.
Let $Y \in \mathbb{R}^{m \times N}$ stack $N$ masked-patch pixel targets of dimension $m$. Let $H \in \mathbb{R}^{d \times N}$ be the free encoder features that the decoder reads for those targets. Let $W \in \mathbb{R}^{m \times d}$ be a linear decoder, and let $P$ be a one-hot matrix giving each target's grid location, read by a decoder-side position table $B$. The disentangled model solves
$$\min_{W,H,B}\ \|Y - WH - BP\|_F^2 + \lambda\left(\|W\|_F^2 + \|H\|_F^2\right),$$
where $\lambda$ is the weight-decay strength. In the entangled model, $B$ is removed, so position can reach the output only through $H$.
The analysis rests on a standard fact about weight-decayed factorizations: the product $WH$ at the optimum keeps the singular directions of its target, with singular values soft-thresholded to $(\sigma_k - \lambda)_+$. This holds because the penalty is equivalent to a nuclear-norm penalty on $WH$. In the entangled model the target is $Y$ itself. Its top singular directions include the per-location mean patch and low-frequency layout, which have large $\sigma_k$ in natural images. In the disentangled model the target is the residual $Y - BP$, so all $d$ feature directions go to content.
The model therefore predicts a measurable crowding effect: the number of surviving content directions should increase by roughly the number of position-dominated singular values above $\lambda$. We first verify this in the exact model on image patches. We then check whether the position-decodable variance fraction of real ViT encoders follows the same ordering.
This toy model has known limitations. It ignores attention, cross-patch mixing and LayerNorm. Neural collapse versus low-rank bias: Is deep neural collapse really optimal? and The Implicit Bias of Depth: From Neural Collapse to Softmax Codes show that depth and weight decay add a low-rank pressure the shallow model omits. We treat the model as a source of predictions about rankings across conditions, not as a proof about ViTs. That low-rank pressure would make crowding worse, not better, which is why we expect the ranking to hold.
Research Program
The work has three parts.
The first part is the theory above, computed exactly on ImageNet patch statistics. For each $\lambda$ and $d$ it produces a predicted fraction of feature energy assigned to position. That fraction is the number the real experiments must match in ordering.
The second part is a set of masked-autoencoder encoders that differ only in how position enters. The control is the plain MAE ViT, which uses fixed additive 2D sinusoidal embeddings, from Masked Autoencoders Are Scalable Vision Learners. The first treatment adapts DeBERTa-style disentangled attention to 2D. Its scores combine three terms: content-to-content, content-to-relative-position and relative-position-to-content, with relative offsets on the 2D patch grid. The decoder adds absolute positions just before predicting pixels, following DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION. The second treatment is a cheaper variant with no position vectors in the residual stream. It adds a learned logit bias, indexed by relative 2D offset, to each attention score, using the gate-based parameterization that Mask-Based Priors Are More Persistent than Query-Key Initializations shows resists being washed out by gradients. We learn this bias end to end instead of hand-designing it. That removes that paper's oracle assumption in the one setting where translation structure makes a learned offset table natural.
The third part is a measurement suite applied identically to every frozen encoder. It has four components:
- Position decodability. The fraction of per-token feature variance explained by a linear map from patch coordinates.
- Effective rank. The effective rank of token and pooled features.
- Directional collapse. Decision-axis class-distance-normalized variance, as defined in Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning. That paper measured this quantity post hoc on frozen MAE and other self-supervised models but did not explain what produces it. Our claim is that position routing is one controllable factor.
- Depth trend. Collapse metrics across encoder depth on small-scale runs, so that we can check whether the depth trend reported by Súkeník et al. carries over to reconstruction objectives.
Other work in this area differs from ours in specific ways. Information Flow in Self-Supervised Learning attributes MAE's weak probing to low effective rank and fixes it with a loss term; it does not ask which signal occupies the rank. RevColV2: Exploring Disentangled Representations in Masked Image Modeling separates low-level from semantic information with reversible columns, at a reported 1.21x inference slowdown. Its "disentanglement" works across whole network columns and leaves position mixed into features inside attention. Decoupling The "What" and "Where" With Polar Coordinate Positional Embedding shows that separating content from position fixes positional reasoning in sequence models. It does not study vision or feature geometry.
Baselines
The naive reference point is vanilla MAE with additive positional embeddings, trained with the identical recipe. The file reports that M-MAE's ViT-Base linear probe exceeds vanilla MAE by about 7 points (Information Flow in Self-Supervised Learning). We will reproduce vanilla MAE ourselves rather than rely on that implied number. Vanilla MAE is the control on which every geometry metric is anchored.
The strongest existing competitor on the linear-probing question is M-MAE. It adds a total-coding-rate matrix-entropy term to the MAE loss and reports 62.4% ImageNet-1K linear-probe top-1 on ViT-Base, against 58.5% for its U-MAE baseline (Information Flow in Self-Supervised Learning). It is the right comparison because it attacks the same symptom, low feature rank, without changing the architecture. We will rerun it at our epoch budget. Its entropy coefficient will receive the same number of tuning trials as our only new hyperparameter, the decoder-side position scale. Beating it means a higher linear probe at equal pretraining FLOPs and equal tuning trials. If the probes are within run-to-run noise, our method must also show the predicted geometry change that M-MAE does not.
RevColV2 is the strongest competitor for "disentangled MIM" overall. It reports 88.4% ImageNet-1K top-1 after ImageNet-22K intermediate fine-tuning and 60.4 mIoU on ADE20K. We cannot match that pipeline's compute. We therefore compare on ImageNet-1K-only fine-tuning and on latency per image. Success here does not mean beating 88.4%. It means recovering some of RevColV2's semantic benefit at close to plain-ViT latency, which we will measure directly.
Two ablations isolate the key ingredient. The first keeps the disentangled attention but also adds absolute position into the encoder input. If this arm loses the gains, the effect comes from position living in the features, not from extra attention parameters. The second matches the extra parameters and FLOPs of the disentangled arm by widening the plain encoder. This rules out an explanation based purely on added capacity.
Why not just…?
Why not just project position out of frozen MAE features before probing? This is the most serious threat, and we run it. For each frozen vanilla-MAE encoder we fit the coordinate-to-feature linear map, remove the position-predictable subspace from every token, and then pool and probe. If this post-hoc projection recovers most of the treatment arm's gain, the problem lies in how features are read out, not in what was learned. The paper's claim would then shrink to a cheap diagnostic, and we would report it that way. The theory predicts that projection should help only partly. Under crowding, the semantic directions displaced by position were never learned in the first place, so removing position afterward cannot restore them.
Why not just add the matrix-entropy regularizer? M-MAE raises effective rank but is blind to which signal fills that rank. Its authors also note that the coefficient is sensitive and needs retuning between model sizes. Our measurement suite will show whether M-MAE's extra rank goes to position-decodable directions or to class-relevant ones. We also run the combination. If position routing and the entropy term help additively, they address different causes. If they do not add, M-MAE is already achieving what we do, and our contribution reduces to the explanation.
Why not just use a relative position bias and skip the theory? We agree the mechanism may be this simple, and the relative-bias arm is in the study for that reason. The theory's role is to predict which simple changes should work: any design that keeps position out of the residual stream. It also predicts by how much, through the position-decodable variance fraction. If relative bias matches full disentangled attention, that is the predicted outcome, and it is the cheaper recommendation, avoiding the roughly 30% attention overhead reported for DeBERTa.
Why not just fine-tune, since MAE already fine-tunes well? Frozen features matter for few-shot transfer and for sharing one backbone across many tasks. Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning ties exactly these uses to decision-axis variance, the quantity we predict will improve. We still report full fine-tuning so that any probing gain is not bought with a fine-tuning loss.
Proposed Experiments
Small-scale study. We pretrain ViT-Tiny and ViT-Small masked autoencoders on CIFAR-10 and CIFAR-100 at 75% masking, sweeping encoder depth from 4 to 24 blocks and weight decay over a grid. For each run we record:
- position-decodable variance;
- effective rank;
- decision-axis variance;
- linear probe accuracy.
Each run is compared against the exact feature-model prediction computed from the same patch statistics.
Main study. We pretrain ViT-Base on ImageNet-1K for a fixed epoch budget, identical across all arms: vanilla, M-MAE, disentangled attention, relative bias, the two ablations, and the combination with M-MAE. We report:
- linear probe top-1;
- fine-tuning top-1;
- few-shot accuracy on mini-ImageNet;
- decision-axis orthogonality across attribute tasks on CelebA and CUB-200;
- ADE20K mIoU;
- inference latency.
Every number comes from three seeds.
Decisive checks. The hypothesis is confirmed only if three things hold together. First, the position-decodable variance fraction drops sharply in the arms that keep position out of the residual stream, and comes back in the ablation that re-adds it. Second, across all arms and depths, linear-probe accuracy and decision-axis variance follow that fraction monotonically, as the feature model predicts. Third, post-hoc projection on vanilla MAE recovers clearly less than the architectural change.
The hypothesis is refuted if any of the following occurs: vanilla MAE features turn out to carry little position-decodable variance; removing it does not move the probe; or projection alone matches the treatment. In any of those cases the MAE probing gap must have another cause, and the rank-based explanation behind M-MAE becomes the better-supported account.
References
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
- Neural collapse versus low-rank bias: Is deep neural collapse really optimal?
- The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
- Masked Autoencoders Are Scalable Vision Learners
- DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION
- Mask-Based Priors Are More Persistent than Query-Key Initializations
- Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning
- Information Flow in Self-Supervised Learning
- RevColV2: Exploring Disentangled Representations in Masked Image Modeling
- Decoupling The "What" and "Where" With Polar Coordinate Positional Embedding