Leakage as a Dose: Measuring How Much Convolutional Bleed Masked Image Modeling Can Tolerate

Masked image modeling teaches a vision network by hiding most of an image and asking it to reconstruct the missing parts. Convolutional networks have an awkward relationship with this task. A convolution is a small sliding filter, and when it straddles the edge of a hidden region it copies visible pixel information into the hidden one. Stack enough layers and the hidden region fills in by local smoothing, so the network can "solve" the task without understanding the scene. Every leading method treats this leakage as all-or-nothing and forbids it completely, using sparse convolutions that compute only at visible locations or masked convolutions that re-zero hidden locations after every layer. The main evidence for that rule is a single binary ablation: removing masked convolutions drops ImageNet fine-tuning accuracy from 84.6% to 81.5% in MCMAE. That ablation also changes what the normalization layers see, and it never asks whether a little leakage is harmless or even useful.

We propose to treat leakage as a continuous, controllable dose. The mechanism is a dense, hardware-native operator: an ordinary convolution followed by multiplication with a mask that is grown outward by a chosen number of pixels. This lets the encoder bleed visible information exactly $R$ input pixels into each hidden patch and no further. Sweeping $R$ from zero (equivalent to sparse convolution) to unbounded (plain dense convolution) gives a dose–response curve for representation quality. We cross this sweep with mask patch size, and we separate the shortcut effect of leakage from its side effect on normalization statistics. The deliverable is an empirical law that says when leakage hurts. It also tests a specific hypothesis: the harm depends on the ratio of leakage distance to mask patch size, so a nonzero, size-matched leakage budget can match or beat strict isolation on dense transfer tasks, with no sparse library required.

Core Claim

The claim is that convolutional leakage harms masked pretraining only once it reaches deep into the hidden region. A thin band of leakage at mask borders leaves the patch interior as a real semantic prediction problem. We predict the interior band carries most of the learning signal, and the border band may reduce the gap between pretraining, where images have holes, and fine-tuning, where images are whole. If this holds, the field's "zero leakage" rule is a conservative special case of a tunable design choice.

If it fails, meaning any $R>0$ degrades results monotonically even when $R$ is far smaller than the patch size, we will have the first controlled evidence that strict isolation is required. We will also know that the 3.1-point drop in MCMAE comes from the shortcut itself and not from normalization side effects. Either outcome is informative, and the second still settles a question that current papers only assert.

How Prior Work Frames the Problem

Several papers motivate the proposal by documenting leakage as a failure mode. SparK names "mask pattern vanishing": dense 3×3 kernels spread visible features into hidden positions until the mask boundary disappears in deep stages. VideoMAC reports the same effect, "mask dissipation," for 3D video convolutions. Three Things Everyone Should Know About Vision Transformers shows that even a convolutional stem on a transformer leaks across patch borders and weakens BEiT-style pretraining. All three describe leakage qualitatively and respond by eliminating it. None measures how far the bleed must travel before it hurts.

Other papers supply the method we reuse, which is isolation of visible features. SparK, the fully convolutional masked autoencoder of ConvNeXt V2, GreenMIM, and HySparK compute only at visible coordinates using sparse convolutions. MCMAE and its companion ConvMAE restrict 5×5 depthwise kernels to visible pixels and share one mask across scales. Our $R=0$ endpoint reproduces this family with dense operations. Our contribution is everything between $R=0$ and $R=\infty$.

A third group sidesteps spatial holes entirely and serves as a constraint and comparison. Corrupted Image Modeling fills hidden patches with generator-synthesized pixels so that an unmodified ResNet-50 sees natural-looking input, at 1.8–2.6× the per-epoch cost. Masked Frequency Modeling masks Fourier components rather than pixels. Both give up skipping hidden regions. Both also implicitly argue that the pretrain–fine-tune input mismatch matters, which is the same mismatch a small leakage band might soften.

Finally, ConvNeXt V2 finds that isolated masked training causes channel collapse, meaning many channels become inactive, and fixes it with Global Response Normalization (GRN), a layer that rescales channels by their spatial L2 norm. Because GRN pools over space, leaked positions change its statistics. This makes GRN a confound we must control rather than ignore.

Research Program

The operator works as follows. Let $M_\ell$ be the binary visible-mask at the resolution of layer $\ell$, obtained by upsampling the patch mask as in MCMAE. After each convolution we multiply the output by $D_{r_\ell}(M_\ell)$, where $D_r$ dilates the visible region by $r$ pixels at that layer's resolution. The per-layer radii are chosen so that the total leakage distance, measured in input pixels, never exceeds a budget $R$:

$$\sum_{\ell} r_\ell \cdot s_\ell \le R,$$

Here $s_\ell$ is the downsampling stride at layer $\ell$. Leakage is front-loaded until the budget is spent, after which the mask is frozen. The setting $R=0$ is masked or sparse isolation, and removing the mask entirely gives $R=\infty$. Everything is dense tensor arithmetic, so it runs on any accelerator without SpConv or MinkowskiEngine. That is a practical side benefit and not the main claim.

The first analysis separates the shortcut from normalization. In a BatchNorm network (ResNet-50), statistics computed over all positions include hidden zeros or leaked values. We therefore run each $R$ twice: once with ordinary normalization, and once with "visible-and-leaked-only" statistics that exclude still-hidden positions. ConvNeXt V2 uses per-position LayerNorm and is immune to this issue, but its GRN pools spatially, so we run it with GRN pooling over all positions and with GRN pooling over active positions only. If MCMAE-style drops vanish under masked statistics, the harm was normalization contamination and not information leakage.

The second analysis is a direct measurement of the shortcut. For each pretrained model we plot reconstruction error inside hidden patches as a function of distance from the nearest visible pixel. A leakage shortcut should appear as near-zero error in a band of width roughly $R$ and normal error beyond it. A healthy model shows a smoother profile. We pair this with a representation-level check: linear-probe accuracy computed separately from features pooled over border bands and over interior regions of hidden patches. Together these show whether the interior still carries the semantic signal.

The third analysis is the core grid. We vary $R \in \{0, 4, 8, 16, 32, \infty\}$ against mask patch size $P \in \{16, 32, 64\}$ at a fixed 60% mask ratio, the ConvNeXt V2 default. Our hypothesis predicts that representation quality depends mainly on $R/P$, with a flat region for $R/P$ up to about 0.25 and degradation beyond about 0.5. The reasoning is that leakage of distance $R$ trivializes a band of area roughly $4RP$ in a $P\times P$ patch, so the fraction of hidden area still requiring inference falls roughly as $1-4R/P$.

Baselines

The naive reference point is a plain dense ConvNet trained with zero-filled holes and no masking after convolutions ($R=\infty$). It shows the full cost of leakage in our setup. We can sanity-check our implementation against the direction reported by MCMAE, whose 84.6% with masked convolutions versus 81.5% without is the only reported leakage ablation in the file. A second reference is supervised training from scratch with the same fine-tuning recipe.

The strongest existing method on this question is the isolation approach of ConvNeXt V2. It is the best-scaling pure-ConvNet masked pretraining in the file and is architecturally identical to our backbone, which makes it the cleanest comparison. ConvNeXt V2 reports 76.7% ImageNet-1K top-1 for the 3.7M-parameter Atto model. We will first reproduce this point with our dense $R=0$ operator under the official recipe. Only then will we report $R>0$ variants trained with identical data, epochs, augmentation, fine-tuning schedule, and the same small learning-rate search for every arm. Beating it means a nonzero $R$ that improves ImageNet fine-tuning by more than seed variance (three seeds), or that improves COCO box AP or ADE20K mIoU by at least 0.5 points while staying within 0.2 points on ImageNet. Ties count as a ties, not a wins. For the ResNet-50 track, SparK is the isolation baseline. Corrupted Image Modeling (80.7% ResNet-50) and Masked Frequency Modeling (78.5% ResNet-50) are dense-input alternatives that we cite rather than rerun, since their recipes differ.

The key ablation isolates the leakage budget from its normalization side effect, using the masked-statistics variants described above, at matched $R$. A further ablation keeps $R$ fixed but scatters leakage randomly in place of a contiguous border band. This tests whether any benefit comes specifically from geometrically coherent boundary context.

Why not just…?

Why not just use strict isolation, since it already works? Strict isolation is the $R=0$ point of our sweep, and if it wins everywhere we will say so. What it does not provide is any evidence that it is optimal. The only quantitative support for strict isolation is a binary ablation in MCMAE, which conflates information leakage with normalization contamination. Our masked-statistics arms settle whether that 3.1-point drop is really about leakage.

Why not just use bigger mask patches so leakage cannot reach the interior? This is the most serious alternative, because our own hypothesis implies that increasing $P$ and increasing $R$ trade off. The $R\times P$ grid tests this directly. If, at matched $R/P$, simply enlarging patches with $R=\infty$ recovers isolation-level accuracy, then the right recommendation is "choose patch size" and not "tune leakage." In that case we will report the result in those terms. We expect it to fail for deep networks, because unbounded dense leakage eventually crosses any finite patch, but the experiment decides this.

Why not just fill the holes with plausible pixels, as in Corrupted Image Modeling? That approach removes artificial boundaries but costs 1.8–2.6× per epoch and processes every pixel (Corrupted Image Modeling). Our operator costs one mask multiplication per layer. If a small $R$ captures the boundary-smoothing benefit CIM gets from realistic fills, we obtain it nearly for free. We will compare reported ResNet-50 numbers while noting recipe differences.

Why not just add GRN and assume collapse, not leakage, is the real problem? ConvNeXt V2 shows collapse appears under isolation. Leakage could plausibly reduce or worsen it. We will count dead channels (those with near-zero activation variance) at every $R$, with and without GRN, so that collapse and leakage effects are not confused.

Proposed Experiments

All pretraining uses ImageNet-1K. The full grid runs at ConvNeXt V2-Atto scale for 200 epochs and on ResNet-50 with a SparK-style decoder, which keeps the roughly 36-arm sweep affordable. The best two $R$ values plus the $R=0$ and $R=\infty$ arms are then rerun at ConvNeXt V2-Tiny for 800 epochs to check that the trend survives scale.

Evaluation covers four things. The first is ImageNet-1K fine-tuning top-1 and linear-probe accuracy. The second is COCO 2017 Mask R-CNN box and mask AP, compared against ConvNeXt V2's own transfer setting. The third is ADE20K UperNet mIoU; the file reports 51.7 for MCMAE-Base but gives no ADE20K number for ConvNeXt V2, so we will establish our own $R=0$ reference. The fourth is the reconstruction-error-versus-distance profiles.

Three outcomes are decisive. First, if the masked-statistics BatchNorm arm at $R=\infty$ closes most of the gap to $R=0$, the community's leakage explanation is largely a normalization artifact. Second, if results collapse onto a single curve in $R/P$ with a flat region above zero, leakage is a dose with a safe budget, and a dense operator with no sparse dependency suffices. Third, if any $R>0$ in that flat region gives a consistent dense-transfer gain over isolation across three seeds, partial leakage is a useful design choice and not just a tolerated one. If none of these hold, we publish the curve as calibrated evidence that strict isolation is required at every scale we tested.