Masked image modeling trains a vision transformer by hiding most image patches and asking it to fill them in. Its loss is computed only at the hidden positions. Nothing in the objective asks the network to keep the fine detail of the patches it can see: the exact position of an edge inside a 16×16 patch, a thin vessel, a texture boundary. Dense tasks such as segmentation and edge detection later read exactly those visible-patch tokens, and they need exactly that detail. We propose to measure how much sub-patch detail survives to the last layer of masked-pretrained encoders, and to add one auxiliary objective that requires each visible token to remain decodable into its own high-frequency wavelet coefficients. The objective uses a shift-invariant, directional wavelet so the requirement does not depend on where the patch grid happens to fall.
The opportunity matters because the strongest masked pretraining methods are moving away from pixels. A Unified View of Masked Image Modeling (MaskDistill) predicts the features of a semantic teacher such as CLIP at masked positions and reports strong classification and segmentation. Those teacher features were themselves produced by a coarse patch tokenizer, so the student learns to discard sub-patch structure. Work on downsampling loss in convolutional networks shows what discarded detail costs. RED-Net shows that compressed encoders force decoders to blur fine detail. LiftPool and Spectral U-Net show that making downsampling invertible recovers boundary accuracy. Neither line of convolutional work touches self-supervised masked pretraining.
Core Claim
Masked pretraining objectives leave visible-token detail unconstrained, and teacher-feature distillation actively pushes it out. A small per-token loss can preserve it without hurting semantics. The loss reconstructs the visible patch's own band-normalized detail coefficients from the final-layer token, using the Dual-Tree Complex Wavelet Transform (DTCWT): a wavelet decomposition that splits detail into six orientations and is nearly unchanged under small shifts. Such a loss should improve boundary-sensitive dense tasks over pixel-loss and distillation-only pretraining at matched compute. Two ingredients are specific: per-band normalization and shift invariance. Plain visible-patch pixel loss is dominated by low-frequency energy and should not achieve the same effect.
Written out, the pretraining loss for an image $x$ with masked set $M$ and visible set $V$ is
$$\mathcal{L} = \sum_{i \in M} \ell\big(h(s_i), \mathrm{LN}(t_i)\big) + \lambda \sum_{j \in V} \sum_{b \in B} \big\| g_b(s_j) - \mathrm{norm}_b(D_b(x_j)) \big\|^2 .$$
In this loss, $s_i$ is the student's final-layer token at position $i$, $t_i$ is the teacher feature, and $h$ and $\ell$ are MaskDistill's head and Smooth-$\ell_1$ loss. $D_b(x_j)$ gives the DTCWT detail coefficients of visible patch $x_j$ in orientation-and-scale band $b$. $\mathrm{norm}_b$ rescales each band to unit variance across the dataset, so weak high-frequency bands count as much as strong low-frequency ones. Each $g_b$ is a linear map. Keeping $g_b$ linear is deliberate: it forces the detail to be linearly present in the token that downstream heads consume, rather than recoverable only by a deep decoder.
Relation to Prior Work
The closest work is Wavelet-Driven Masked Image Modeling (WaMIM). It reconstructs standard wavelet coefficients of masked regions and assigns high-frequency targets to shallow layers and low-frequency targets to deep layers. That assignment trains the deepest tokens toward coarse structure, which is the opposite of what we want for dense heads. WaMIM also uses the standard discrete wavelet transform, which shifts and aliases with grid position, and the authors leave other wavelet bases open. We target visible tokens at the final layer, with a shift-invariant transform. WaMIM therefore supplies a method we reuse (wavelet targets are cheap and effective) and a baseline we must beat. It reports 83.8% ImageNet-1K top-1 for ViT-B in 96 GPU hours, against MAE's 83.6% in 752 GPU hours.
FreMIM reconstructs global Fourier spectra from masked medical images. It motivates the proposal by showing that frequency supervision sharpens boundaries, with a 1.57% Jaccard gain on ISIC 2018. It also constrains our design: a global spectrum cannot say where an edge is, so we use a localized wavelet. Spectral U-Net supplies the transform, since it uses DTCWT for lossless supervised downsampling. KiU-Net shows that undercomplete encoders erase small structures even when they have skip connections. We take that as motivation, but we do not adopt its costly overcomplete branch. SETR avoids progressive downsampling but still applies a 16× reduction at patch embedding. For ViT-B, that embedding maps 768 pixel values to 768 dimensions and is not necessarily lossy. This is why we locate the loss in what pretraining asks the encoder to keep, not in the patch embedding itself.
Research Program
The first component is a diagnostic. For each layer of publicly comparable ViT-B encoders pretrained with MAE-style pixel targets, MaskDistill-style teacher distillation, and WaMIM-style wavelet targets, we fit frozen linear probes. The probes predict each visible patch's normalized DTCWT detail bands and the sub-patch location of its dominant edge. Plotting retention against depth tells us whether detail is lost gradually, lost at the final layers, or never encoded. The prediction is that distillation shows the steepest late-layer drop and WaMIM the drop next in size, because of its deep-layers-get-low-frequency assignment.
The second component is the objective above, added to a MaskDistill-style student. All variants use the same teacher, a ViT-B student, and 300 ImageNet-1K epochs. We sweep $\lambda$ over a small grid on a held-out pretraining validation set.
The third component is analysis. We shift test images by sub-patch offsets of 1 to 15 pixels and measure how much the final-layer tokens and dense predictions change. This tests whether DTCWT targets buy grid robustness that standard wavelet targets do not.
Baselines
The naive reference is the MaskDistill-style student with no auxiliary loss, trained with identical data, epochs, teacher, and fine-tuning recipes. The file reports 85.5% ImageNet-1K top-1 for MaskDistill ViT-B in A Unified View of Masked Image Modeling. Its 58.8 ADE20K mIoU is for ViT-H, and the file has no reported ViT-B ADE20K result. We will reproduce the ViT-B number in our own pipeline and treat that reproduction as the reference. Beating it means a boundary-metric gain on ADE20K and edge benchmarks at no more than 0.3 points of top-1 loss.
The strongest existing detail-aware competitor is WaMIM, retrained at the same epoch budget with its released recipe. On ADE20K and COCO the file gives only relative claims: COCO AP 0.5 above SimMIM and 0.9 above GreenMIM. There is no reported result found on Lacuna for WaMIM ViT-B ADE20K mIoU or for any masked-pretrained backbone on BSDS500 edges. Both must therefore be run by us. Beating WaMIM means higher edge F-scores and higher boundary IoU on ADE20K with comparable classification accuracy.
The ablation that isolates the key ingredient replaces our band-normalized DTCWT target with a raw-pixel reconstruction of the visible patch, given the same weight budget and the same linear head. A second ablation swaps DTCWT for the standard discrete wavelet transform to test whether shift invariance matters. A third moves the detail loss to a shallow layer, WaMIM-style, to test whether the final layer is the right place for it.
Why not just…?
Why not just add a pixel reconstruction loss on the visible patches too? This is the most serious threat, and it is run as the central ablation rather than argued away. Our reason to expect it to fall short is that natural-image pixel error is dominated by low-frequency energy. A token can drive that loss low while still smearing edge position, and WaMIM's evidence that structured frequency targets outperform raw pixels supports this reading. If the pixel variant matches ours on edge and boundary metrics, the contribution reduces to the diagnostic, and we will report it that way.
Why not just give the dense head skip connections to shallow layers or the raw image? RED-Net shows that skips restore detail in restoration networks. A raw-image stem skip may close much of the edge-detection gap after full fine-tuning. Against this, Spectral U-Net beats nnU-Net, which already has skips, by 1.28% and 2.35% Dice on two retinal fluid classes. That result suggests skips do not replace preserved representations. We settle the question empirically: every dense comparison is run both with a UPerNet-style multi-layer head plus a high-resolution stem skip, and with a frozen backbone and a linear dense probe. If our gain vanishes under the skip head after full fine-tuning, we will say so. The claim would then narrow to frozen-backbone and low-label regimes.
Why not just use smaller patches? Smaller patches quadruple sequence length at 8×8 and make attention cost grow quadratically, as SETR notes. We include a ViT-B/8 distillation baseline at matched GPU hours, not matched epochs, to see whether compute spent on resolution beats compute spent on our loss.
Proposed Experiments
Pretraining is on ImageNet-1K, with fine-tuned top-1 reported to confirm semantics are not sacrificed. Dense evaluation covers ADE20K mIoU plus boundary IoU, Cityscapes mIoU, and edge detection on BSDS500 and NYUDv2 using ODS, OIS, and average crispness, following DiffusionEdge. Thin-structure medical transfer uses ISIC 2018 Jaccard and GLAS Dice, comparing against FreMIM-style pretraining on the same data.
Four outcomes decide the proposal. First, the diagnostic must show measurable final-layer detail loss in distillation-pretrained encoders; if it does not, the premise fails. Second, our loss must raise probe retention and edge ODS over both the visible-pixel ablation and WaMIM. Third, the gain must survive at least the frozen-backbone protocol when skip-connected heads are present. Fourth, DTCWT must reduce prediction variance under sub-patch shifts relative to the standard wavelet transform. All runs use three seeds, and differences smaller than the seed spread are reported as null.
References
- A Unified View of Masked Image Modeling
- Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections
- LiftPool: Bidirectional ConvNet Pooling
- Spectral U-Net: Enhancing Medical Image Segmentation via Spectral Decomposition
- Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation
- FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentation
- KiU-Net: Towards Accurate Segmentation of Biomedical Images Using Over-Complete Representations
- Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers
- DiffusionEdge: Diffusion Probabilistic Model for Crisp Edge Detection