Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation

Wenzhao Xiang, Chang Liu, Hongyang Yu, Xilin Chen

Abstract

Masked Image Modeling (MIM) has emerged as a powerful paradigm for self-supervised visual representation learning. However, standard pixel-based reconstruction methods often suffer from high computational costs due to data redundancy and a bias toward fine-grained textures. We present WaMIM (Wavelet-Driven Masked Image Modeling), a novel approach that utilizes multi-level wavelet coefficients as reconstruction targets. By leveraging the Discrete Wavelet Transform (DWT), WaMIM achieves a hierarchical alignment between encoder layers and frequency bands, significantly accelerating pre-training. Empirical results on ImageNet-1K demonstrate that WaMIM can achieve 83.8% top-1 accuracy with only 13% of the computational budget required by standard methods like MAE.

Introduction

Masked Image Modeling (MIM) has become a cornerstone of self-supervised learning, where models learn by "filling in the blanks" of a masked image. However, most popular methods, such as Masked Autoencoders (MAE), reconstruct raw pixels. This approach faces two primary challenges. First, raw pixels contain massive amounts of redundant information, leading to inefficient training. Second, pixel-based reconstruction forces the model to focus on fine-grained textures and high-frequency noise, which are often secondary to high-level semantic understanding.

While frequency-domain methods like Fourier Transforms have been explored to reduce redundancy, they often lose spatial localization—the ability to identify where a specific frequency component exists in an image. WaMIM addresses these issues by using wavelets to provide a representation that is compact, frequency-aware, and spatially preserved.

Methodology: The WaMIM Framework

WaMIM leverages the Discrete Wavelet Transform (DWT) to perform multi-resolution analysis, capturing both frequency and spatial information simultaneously. The pipeline consists of several key stages:

  1. Multi-level Decomposition: The input image is decomposed into several levels of wavelet coefficients representing different scales and frequencies.
  2. Hierarchical Target Assignment: The model employs a "shallow-to-deep" strategy. Shallow encoder layers are tasked with reconstructing high-frequency wavelet targets (fine details like edges), while deeper layers reconstruct low-frequency targets (global shapes and structure).
  3. Lightweight Decoders: To minimize overhead, WaMIM uses thin decoders—often just a single Transformer block—for each reconstruction level.
  4. Adjustable Loss Weighting: Different weights are assigned to the reconstruction loss of each level, allowing the model to prioritize information-dense scales.
The WaMIM pipeline: Input images are decomposed via DWT into multi-level wavelet coefficients, which serve as reconstruction targets for a hierarchical encoder-decoder architecture.

Experimental Results

WaMIM was validated across standard benchmarks, including ImageNet-1K for classification, COCO for object detection, and ADE20k for semantic segmentation. The results highlight a remarkable efficiency-to-performance ratio.

Efficiency Gains

On the ViT-B architecture, WaMIM achieved 83.8% top-1 accuracy on ImageNet-1K with only 96 GPU hours (400 epochs). In contrast, the original MAE requires 752 GPU hours (1600 epochs) to reach a comparable 83.6%. This represents a ~7.8x speedup. Similar gains were observed for the Swin-B architecture, where WaMIM consistently outperformed methods like SimMIM and GreenMIM in terms of accuracy per training hour.

Qualitative visualization of multi-level wavelet decomposition (left) and quantitative comparison of Top-1 accuracy vs. GPU hours for ViT-B and Swin-B architectures (right).

Downstream Performance

Beyond classification, WaMIM demonstrates strong transferability. In COCO object detection tasks, WaMIM outperformed SimMIM and GreenMIM by 0.5 and 0.9 AP points, respectively, despite significantly shorter pre-training durations. For smaller models like ViT-S, WaMIM reached 82.0% accuracy, setting a new efficiency benchmark for that scale.

Conclusion and Future Work

WaMIM proves that frequency-domain targets, specifically wavelets, are a natural fit for the hierarchical nature of neural networks. By avoiding the "pixel-pushing" bottleneck, the model learns underlying structures faster and more effectively.

Future research directions include exploring different wavelet bases (such as Daubechies) for specialized imagery and implementing automated, learnable loss weighting to further optimize the training trajectory. Additionally, refining masking strategies to better match multi-scale target resolutions remains an area for implementation improvement.

Summary updated

Abstract

Masked Image Modeling (MIM) has garnered significant attention in self-supervised learning, thanks to its impressive capacity to learn scalable visual representations tailored for downstream tasks. However, images inherently contain abundant redundant information, leading the pixel-based MIM reconstruction process to focus excessively on finer details such as textures, thus prolonging training times unnecessarily. Addressing this challenge requires a shift towards a compact representation of features during MIM reconstruction. Frequency domain analysis provides a promising avenue for achieving compact image feature representation. In contrast to the commonly used Fourier transform, wavelet transform not only offers frequency information but also preserves spatial characteristics and multi-level features of the image. Additionally, the multi-level decomposition process of wavelet transformation aligns well with the hierarchical architecture of modern neural networks. In this study, we leverage wavelet transform as a tool for efficient representation learning to expedite the training process of MIM. Specifically, we conduct multi-level decomposition of images using wavelet transform, utilizing wavelet coefficients from different levels to construct distinct reconstruction targets representing various frequencies and scales. These reconstruction targets are then integrated into the MIM process, with adjustable weights assigned to prioritize the most crucial information. Extensive experiments demonstrate that our method achieves comparable or superior performance across various downstream tasks while exhibiting higher training efficiency.
BibTeX
@inproceedings{xiang2025wavelet,
  author = {Wenzhao Xiang and Chang Liu and Hongyang Yu and Xilin Chen},
  title = {{Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32930},
}
Figure