Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates

Cong Xie, Sanmi Koyejo, Indranil GuptaarXiv 2019 · arXiv · OpenReview

Abstract We consider distributed on-device learning with limited communication and security requirements. We propose a new robust distributed optimization algorithm with efficient communication and attack tolerance. The proposed algorithm has provable convergence and robustness under non-IID settings. Empirical results show that the proposed algorithm stabilizes the convergence and tolerates data poisoning on a small number of workers.

Introduction

There is a massive push to move machine learning training from centralized cloud servers to edge devices such as smartphones, wearables, and IoT sensors. This shift preserves user privacy and leverages on-device data. However, this "federated" or distributed on-device setting introduces severe engineering and security hurdles:

  1. Limited Communication: Edge devices often have constrained bandwidth and cannot afford to constantly stream high-speed updates to a central server.
  2. Non-IID Data: Data on individual devices is typically not independently and identically distributed (non-IID). For example, one user's photo library looks very different from another's, breaking the mathematical assumptions of many standard ML algorithms.
  3. Untrusted Workers: The central server cannot verify the integrity of the data on anonymous edge devices. Malicious users can execute "data poisoning" attacks (e.g., flipping labels) to upload harmful models and ruin the global training process.

Existing methods like Federated Averaging (FedAvg) save communication but lack theoretical convergence guarantees for non-IID data and fail catastrophically if even a few workers are malicious.

The SLSGD Algorithm

To address these challenges, we introduce SLSGD (Secure Local Stochastic Gradient Descent). This algorithm modifies standard distributed training with three key mechanisms designed for robustness and efficiency. The mathematical framework for SLSGD relies on the notation summarized in Table 1.

Table 1: Summary of notation and terms used in the SLSGD framework.

1. Local SGD

Instead of communicating with the server after every single gradient step, devices perform multiple local training iterations ($H_t^i$). They only push their updated model weights to the server periodically, significantly reducing communication overhead.

2. Secure Aggregation via Trimmed Mean

When the server collects models from a random subset of workers ($S_t$), it avoids simple averaging, which is easily skewed by a single extreme value from a poisoned worker. Instead, SLSGD uses a coordinate-wise trimmed mean. It sorts the incoming model parameters, discards the $b$ highest and lowest values, and averages the remainder. This filters out malicious outliers.

3. Moving Average Update

Local updates on non-IID data can cause models to drift, and the trimmed mean introduces extra variance. To counteract this, the server updates the global model using a moving average with weight $\alpha$. It blends the previous global model with the newly aggregated local models to smooth out errors and stabilize convergence.

Experimental Evaluation

We evaluated SLSGD on two standard benchmarks, artificially partitioned across 100 simulated edge devices:

  • CIFAR-10: Image classification using a Convolutional Neural Network (CNN).
  • WikiText-2: Language modeling using an LSTM network.

We tested on both balanced partitions and highly unbalanced partitions (where some devices might only have images of a single class) to simulate realistic, non-IID edge environments. We specifically focused on "label-flipping" attacks, where malicious workers intentionally mislabel data to sabotage the model.

Key Insights and Results

Robustness Against Attacks

Under a label-flipping attack where 2 to 4 out of 10 active workers send poisoned data, the standard FedAvg baseline completely diverges, failing to learn anything useful. In contrast, SLSGD successfully converges and stabilizes even when up to 40% of the active workers in a given round are poisoned.

Theoretical and Empirical Synergy

Unlike prior work that assumes perfectly distributed data, SLSGD provides mathematical proofs of convergence that hold under realistic edge conditions (non-IID data, heterogeneous local iteration counts). Counterintuitively, doing more local iterations before syncing doesn't just save bandwidth; we prove theoretically and show empirically that it actually accelerates the overall convergence of the global model.

Manageable Trade-offs

In safe environments without attacks, robust aggregation adds a small amount of variance, slightly slowing convergence compared to standard FedAvg. However, in hostile or highly unbalanced environments, tuning the moving average weight ($\alpha$) and the trim parameter ($b$) allows the model to reliably converge where standard methods fail.

Conclusion and Future Work

The primary limitation of SLSGD's robust aggregation is mathematical: the trimmed mean requires that strictly less than half of the active workers in any given round are poisoned ($2q < k$). If attackers control the majority of the devices selected for an update epoch, the defense breaks down. Future work will extend this theoretical analysis to cover other threat models, such as random hardware or software failures, beyond adversarial data poisoning.

Summary updated

Abstract

We consider distributed on-device learning with limited communication and security requirements. We propose a new robust distributed optimization algorithm with efficient communication and attack tolerance. The proposed algorithm has provable convergence and robustness under non-IID settings. Empirical results show that the proposed algorithm stabilizes the convergence and tolerates data poisoning on a small number of workers.
BibTeX
@misc{xie2019practical,
  author = {Cong Xie and Sanmi Koyejo and Indranil Gupta},
  title = {{Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates}},
  year = {2019},
  eprint = {1903.06996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.06996},
}
Figure