On the Robustness of Distributed Machine Learning Against Transfer Attacks

Sébastien Andreina, Pascal Zimmer, Ghassan KarameAAAI 2025 · DOI

Introduction

Adversarial attacks represent a significant vulnerability for deep learning models. Among these, transfer attacks are particularly potent; an adversary trains a surrogate model to generate adversarial examples that are then transferred to a target model. This threat model is realistic because it does not require direct access to the target's internal weights. While ensemble and federated learning have been studied for robustness individually, this research explores the security synergy of distributing both training and inference phases through extreme heterogeneity.

Pure Distributed Learning: Embracing Heterogeneity

The authors propose "pure distributed learning," a framework that intentionally maximizes diversity across multiple dimensions to prevent an attacker from finding a single adversarial direction that fools the majority of the network.

Dimensions of heterogeneity in the proposed distributed learning framework, including architecture, optimization, and hyperparameters.

As shown in the parameter space above, heterogeneity is introduced via four primary channels:

  1. Data Diversity: Each node is trained on a disjoint subset of the total dataset.
  2. Architectural Diversity: Nodes utilize different model families, such as VGG, ResNet, DenseNet, and MobileNet.
  3. Optimization Diversity: Nodes independently select from nine different optimizers and five learning rate schedulers.
  4. Independent Hyperparameter Tuning: Each node performs localized tuning for learning rate, momentum, and weight decay.

Theoretically, this diversity reduces the Gradient Similarity between a surrogate model and the target nodes. If the gradients do not point in the same direction, an adversarial perturbation designed for one model will likely fail to fool the others in the ensemble.

Experimental Evaluation

The approach was evaluated using CIFAR10 and FashionMNIST benchmarks. Data was partitioned across varying numbers of nodes ($N = 3, 5, 7$) using both uniform (IID) and Dirichlet (non-IID) distributions to simulate real-world conditions.

The Robustness-Accuracy Trade-off

A common challenge in adversarial defense is the significant drop in clean task accuracy (CA) when attempting to improve robust accuracy (RA). The proposed distributed approach ($D_p$) achieves a superior Pareto frontier compared to standard ensemble methods (ENS).

Pareto frontier comparing Robust Accuracy (RA) vs Clean Accuracy (CA). The distributed approach ($D_p$, dashed line) significantly outperforms standard ensembles (ENS, solid line).

As illustrated in the figure, $D_p$ provides much higher robustness with minimal impact on clean performance, maintaining an 87% clean accuracy while providing 71% robust accuracy. Against the strong "Common Weakness" (CW) attack, the method improved robust accuracy by 33.6% to 41.2% over standard models. While standard ensembles peaked at approximately 37% robust accuracy, the heterogeneous distributed models reached up to 83%.

Key Insights

  • The Diversity Dividend: Heterogeneity in optimizers, schedulers, and hyperparameters was found to be more critical for robustness than architectural variety alone.
  • Scaling with Nodes: Robustness scales nearly linearly with the system size. Regression analysis indicated a 2.88% boost in robust accuracy for every additional node added to the system.
  • Data Distribution Neutrality: The system proved resilient to data non-IIDness. While uneven data distribution slightly lowered clean accuracy, its impact on robustness was negligible.
  • Aggregation Strategy: "Average voting" (averaging probability vectors) consistently outperformed "Hard voting" (majority label vote) in both clean accuracy and robustness.

Conclusion and Future Directions

This work demonstrates that full heterogeneity in distributed learning creates a powerful defense against transfer attacks by diversifying the loss landscape. Future research should address the computational overhead of training $N$ independent models and investigate the system's limits regarding node saturation and adaptive white-box attacks on larger, more complex datasets like ImageNet.

Summary updated

BibTeX
@inproceedings{andreina2025robustness,
  author = {Sébastien Andreina and Pascal Zimmer and Ghassan Karame},
  title = {{On the Robustness of Distributed Machine Learning Against Transfer Attacks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33688},
}
Figure