Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data

Xuanyu Chen, Nan Yang, Shuai Wang, Dong Yuan

Introduction

Training machine learning models on decentralized, unlabeled data—such as footage from distributed security cameras or user mobile devices—is highly desirable for preserving privacy and ensuring scalability. This demand has driven the rapid rise of Distributed Self-Supervised Learning (D-SSL). However, real-world distributed data is rarely uniform; it is typically "non-IID" (non-independent and identically distributed), meaning the data distribution on one client differs significantly from another.

While various algorithms have been proposed to mitigate the performance degradation caused by non-IID data, there remains a glaring lack of theoretical understanding regarding this problem. Foundational questions remain unanswered: Is Federated Learning (FL) better equipped to handle non-IID data than Decentralized Learning (DecL)? Is Masked Image Modeling (MIM) inherently more robust than Contrastive Learning (CL)? This paper aims to settle these questions mathematically and introduces a novel regularization technique to boost real-world performance.

Theoretical Framework and Methodology

The authors tackle the challenge of non-IID data in two phases: establishing a theoretical foundation and designing a practical algorithm.

To build the theoretical framework, they construct a simplified mathematical model of non-IID data distribution across clients.

Visualization of the theoretical non-IID data distribution model across different clients.

By defining a "Representability Vector," the authors quantitatively assess and bound the feature spaces learned by different D-SSL setups. This allows for a rigorous comparison between MIM and CL, as well as between FL and DecL.

To translate these theoretical insights into practice, the authors propose MAR (Masked Alignment Regularization) loss. Their theory indicates that local MIM encoders tend to drift apart before aggregation. To counteract this, MAR adds an adaptive Maximum Mean Discrepancy (A-MMD) penalty to the standard MIM objective, dynamically aligning local and global representations. It employs a cosine decay schedule to apply stronger alignment early in training—when client divergence is highest—and relaxes it as training progresses.

Key Insights

The theoretical analysis yields several critical insights into the behavior of D-SSL frameworks:

  • MIM is Superior to CL for Heterogeneous Data: Masked Image Modeling (e.g., MAE) is mathematically proven to be fundamentally more robust to non-IID data than Contrastive Learning (e.g., SimSiam). CL relies on aligning augmented views of data, which inherently bakes in the biases of the local client's dataset. Conversely, MIM reconstructs raw pixels, forcing the model to retain more uniform, objective information regardless of local data skew.
  • Connectivity Dictates Robustness: In decentralized networks, where clients communicate with local neighbors rather than a central server, robustness scales directly with average network connectivity. The more neighbors a client interacts with, the less sensitive the overall network is to non-IID data.
  • FL Serves as an Upper Bound for DecL: Because Federated Learning acts as a fully connected graph (every client communicates with the central server, and thus indirectly with all others), FL is theoretically guaranteed to be at least as robust against non-IID data as any Decentralized Learning setup.

Empirical Results

Extensive experiments were conducted using Mini-ImageNet for pre-training, and CIFAR-10, CIFAR-100, and ImageNet for fine-tuning and evaluation. Non-IID conditions were simulated using Dirichlet distributions for label skew and unique client-specific augmentations for feature skew.

The empirical results strongly validate the theoretical claims:

  • Validation of MIM Robustness: Under label non-IID settings on CIFAR-10, MIM (MAE + ViT) suffered only a 1.70% accuracy drop compared to an IID baseline, whereas CL (SimSiam + ViT) experienced a steeper 2.82% drop.
  • Impact of Connectivity: In decentralized setups, increasing average client connectivity from approximately 5 to 10 boosted CIFAR-100 fine-tuning accuracy from 46.62% to 52.28%, closely approaching the FL baseline of 52.72%.
  • State-of-the-Art Performance with MAR Loss: The proposed FedMAR algorithm achieved dominant results. On CIFAR-100, FedMAR with a ResNet-18 backbone reached 70.82% accuracy, significantly outperforming baselines like FedU (38.44%) and FedEMA (40.78%), and edging out Orchestra (70.11%). Utilizing a Tiny-ViT backbone pushed FedMAR's performance even higher to 71.28%.

Limitations and Future Work

While the paper provides a robust theoretical foundation, the mathematical proofs rely on simplified linear embedding functions and specific assumptions about the non-IID data generation process, whereas real-world deep neural networks are highly non-linear. Additionally, MAR loss introduces a slight communication overhead (approximately 1-5% depending on batch size) because clients must transmit masked embeddings to calculate the alignment penalty. Future work will aim to extend the theoretical analysis to incorporate more general mixing operators for decentralized aggregation, moving beyond the basic structural properties assumed in this study.

Summary updated

Abstract

Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theoretical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.
BibTeX
@inproceedings{chen2026understanding,
  author = {Xuanyu Chen and NAN YANG and Shuai Wang and Dong Yuan},
  title = {{Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=c3yt5VSZPQ},
}
Figure