MAGMA: Manifold Regularization for MAEs

Alin Dondera, Anuj Singh, Hadi Jamali Rad

Abstract

MAGMA (Manifold Regularization for MAEs) introduces a novel, "plug-and-play" regularization loss designed to significantly enhance the performance of Masked Autoencoders. By enforcing consistency in how data samples relate to one another across different Transformer layers, MAGMA helps Vision Transformers (ViTs) learn smoother, more semantically coherent representations with negligible computational overhead.

Introduction

Masked Autoencoders (MAEs) have become a cornerstone of self-supervised learning (SSL) because they do not require complex data augmentations or negative pairs. However, they are primarily built on Vision Transformers (ViTs), which lack the inherent inductive biases (like translation invariance) found in Convolutional Neural Networks (CNNs). This often results in a need for massive amounts of labeled data for downstream tasks. MAGMA addresses this by regularizing the internal "manifold" of the network, ensuring that the organization of data remains consistent and smooth as information flows from one layer to the next.

Methodology

The core contribution is the MAGMA loss, a batch-wide, layer-wise regularization term applied during the pre-training phase.

Overview of the MAGMA framework, illustrating how consistency is enforced between internal representations at different layers (k and l).

MAGMA analyzes a batch of images and calculates the similarity of their representations at a reference layer (usually the penultimate layer) using a Radial Basis Function (RBF) kernel. It then applies a penalty if those same images diverge significantly in a subsequent target layer (usually the final layer). By employing a normalized Laplacian matrix, MAGMA regularizes the relationships between different images across the network's hierarchy, rather than just comparing different views of the same image.

Experimental Setup

The authors evaluated MAGMA across four standard computer vision benchmarks to demonstrate its robustness across scales:

  • CIFAR-100: 100 classes of low-resolution images.
  • STL-10: Optimized for unsupervised learning with a large unlabeled set.
  • Tiny-ImageNet: A 200-class downsized version of ImageNet.
  • ImageNet-100: A curated subset used to test scalability.

Results and Discussion

The quantitative improvements are striking, particularly for generative models like MAE. On ImageNet-100, adding MAGMA to a standard MAE increased linear probing accuracy from 58.0% to 69.0%. In extreme low-data regimes (using only 1% of labels), the MAGMA-enhanced model more than tripled the performance of the baseline (1.6% vs 0.5%).

Online accuracy comparison between standard MAE and M-MAE, highlighting the immediate performance boost when MAGMA is "plugged in."

Key Insights

  • Manifold Smoothness: By forcing conceptual similarity to persist across layers, MAGMA helps the model ignore background noise and focus more sharply on central objects.
  • Generative vs. Contrastive: MAGMA provides a massive boost to generative models (MAE) but only marginal gains for contrastive models (SimCLR), which already perform a form of manifold regularization.
  • Efficiency: M-MAE (MAE + MAGMA) maintains the same parameter count as the baseline with only a minimal (1.5% to 2%) drop in throughput.

Conclusion

MAGMA offers a simple, effective way to introduce necessary regularization into ViT-based self-supervised models. While its impact is smaller on CNNs—which already possess structural regularization—it represents a significant step forward for Transformer architectures. Future research may explore applying these layer-wise consistency principles to other domains, such as Natural Language Processing.

Summary updated

BibTeX
@inproceedings{dondera2025magma,
  author = {Alin Dondera and Anuj Singh and Hadi Jamali Rad},
  title = {{MAGMA: Manifold Regularization for MAEs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00670},
}
Figure