Abstract
Masked Image Modeling (MIM) has emerged as a powerful paradigm for self-supervised visual representation learning. However, standard pixel-based reconstruction methods often suffer from high computational costs due to data redundancy and a bias toward fine-grained textures. We present WaMIM (Wavelet-Driven Masked Image Modeling), a novel approach that utilizes multi-level wavelet coefficients as reconstruction targets. By leveraging the Discrete Wavelet Transform (DWT), WaMIM achieves a hierarchical alignment between encoder layers and frequency bands, significantly accelerating pre-training. Empirical results on ImageNet-1K demonstrate that WaMIM can achieve 83.8% top-1 accuracy with only 13% of the computational budget required by standard methods like MAE.
Introduction
Masked Image Modeling (MIM) has become a cornerstone of self-supervised learning, where models learn by "filling in the blanks" of a masked image. However, most popular methods, such as Masked Autoencoders (MAE), reconstruct raw pixels. This approach faces two primary challenges. First, raw pixels contain massive amounts of redundant information, leading to inefficient training. Second, pixel-based reconstruction forces the model to focus on fine-grained textures and high-frequency noise, which are often secondary to high-level semantic understanding.
While frequency-domain methods like Fourier Transforms have been explored to reduce redundancy, they often lose spatial localization—the ability to identify where a specific frequency component exists in an image. WaMIM addresses these issues by using wavelets to provide a representation that is compact, frequency-aware, and spatially preserved.
Methodology: The WaMIM Framework
WaMIM leverages the Discrete Wavelet Transform (DWT) to perform multi-resolution analysis, capturing both frequency and spatial information simultaneously. The pipeline consists of several key stages:
- Multi-level Decomposition: The input image is decomposed into several levels of wavelet coefficients representing different scales and frequencies.
- Hierarchical Target Assignment: The model employs a "shallow-to-deep" strategy. Shallow encoder layers are tasked with reconstructing high-frequency wavelet targets (fine details like edges), while deeper layers reconstruct low-frequency targets (global shapes and structure).
- Lightweight Decoders: To minimize overhead, WaMIM uses thin decoders—often just a single Transformer block—for each reconstruction level.
- Adjustable Loss Weighting: Different weights are assigned to the reconstruction loss of each level, allowing the model to prioritize information-dense scales.
Experimental Results
WaMIM was validated across standard benchmarks, including ImageNet-1K for classification, COCO for object detection, and ADE20k for semantic segmentation. The results highlight a remarkable efficiency-to-performance ratio.
Efficiency Gains
On the ViT-B architecture, WaMIM achieved 83.8% top-1 accuracy on ImageNet-1K with only 96 GPU hours (400 epochs). In contrast, the original MAE requires 752 GPU hours (1600 epochs) to reach a comparable 83.6%. This represents a ~7.8x speedup. Similar gains were observed for the Swin-B architecture, where WaMIM consistently outperformed methods like SimMIM and GreenMIM in terms of accuracy per training hour.
Downstream Performance
Beyond classification, WaMIM demonstrates strong transferability. In COCO object detection tasks, WaMIM outperformed SimMIM and GreenMIM by 0.5 and 0.9 AP points, respectively, despite significantly shorter pre-training durations. For smaller models like ViT-S, WaMIM reached 82.0% accuracy, setting a new efficiency benchmark for that scale.
Conclusion and Future Work
WaMIM proves that frequency-domain targets, specifically wavelets, are a natural fit for the hierarchical nature of neural networks. By avoiding the "pixel-pushing" bottleneck, the model learns underlying structures faster and more effectively.
Future research directions include exploring different wavelet bases (such as Daubechies) for specialized imagery and implementing automated, learnable loss weighting to further optimize the training trajectory. Additionally, refining masking strategies to better match multi-scale target resolutions remains an area for implementation improvement.
Summary updated