Federated Learning with Autotuned Communication-Efficient Secure Aggregation

Kallista A. Bonawitz, Fariborz Salehi, Jakub Konecný, Brendan McMahan, Marco Gruteser

Introduction

Federated Learning (FL) enables decentralized devices to collaboratively train a global model while keeping data localized. To ensure privacy, Secure Aggregation (SecAgg) protocols are employed, allowing a central server to compute the sum of user updates without accessing individual gradients. However, SecAgg necessitates quantizing updates into a fixed integer range. This introduces the "Clipping Range Dilemma": a range that is too narrow clips significant signal, while a range that is too wide increases quantization noise. Since the server cannot inspect individual updates, manually tuning this range is inefficient and often impossible as model dynamics shift during training.

The Autotuning Recipe

This paper proposes an automated method to tune compression parameters by leveraging the statistical properties of high-dimensional data. The approach relies on two primary techniques:

  1. Random Rotation: Users apply a random orthogonal matrix (such as the Walsh-Hadamard Transform) to their updates. This "Gaussianizes" the data distribution, ensuring that regardless of the original gradient shape, the aggregated entries follow a predictable Normal distribution.
  2. Modular Wrapping: Instead of traditional clipping, which truncates values outside a range $[-t, t]$, the authors use modular arithmetic. Values that exceed the boundary "wrap around" the modulus.
Visualization of modular wrapping: the tails of the Gaussian distribution (left) are wrapped back into the central range (right) rather than being discarded.

By observing the histogram of the wrapped aggregate, the server can fit a Wrapped Normal Distribution. This allows the server to estimate the variance of the original, unwrapped data and dynamically adjust the clipping range $t$ for subsequent rounds. This process ensures the system stays within an optimal operating regime without human intervention.

Experimental Evaluation

The researchers evaluated the autotuning framework using the CIFAR-10 image classification dataset with a 100-device simulation and an all-convolutional neural network. The experiments focused on the trade-off between quantization error and wrapping distortion, controlled by the parameter $\alpha$ (the probability of an entry wrapping).

The "Goldilocks" Zone for Accuracy

The results demonstrate that there is an optimal wrapping probability $\alpha$ that maximizes model performance. If $\alpha$ is too high, the data becomes overly distorted by wrapping. Conversely, if $\alpha$ is too low (being "too safe"), the quantization bins become too wide, and rounding errors dominate.

Evaluation accuracy on CIFAR-10 across different wrapping probabilities (alpha). The curves show that intermediate values of alpha (e.g., 0.01 to 0.05) achieve the highest accuracy.

The autotuning method successfully maintains high accuracy even with aggressive 8-bit quantization. By effectively managing the communication-privacy-accuracy trade-off, the system reduces bandwidth requirements per training round while remaining robust to the evolving scale of gradients.

Conclusion and Future Directions

The proposed autotuning method provides a practical solution to the clipping range dilemma in Secure Aggregation. By combining random rotations with modular wrapping, the server can infer necessary statistical properties of the global update while strictly adhering to privacy constraints.

Future work includes testing the approach on more complex, deeper architectures and exploring its robustness in highly non-IID (Independent and Identically Distributed) settings. Additionally, a formal theoretical analysis of how the wrapping probability $\alpha$ influences the convergence rate of stochastic gradient descent in FL remains an open area for investigation.

Summary updated

BibTeX
@inproceedings{bonawitz2019federated,
  author = {Kallista A. Bonawitz and Fariborz Salehi and Jakub Konecný and Brendan McMahan and Marco Gruteser},
  title = {{Federated Learning with Autotuned Communication-Efficient Secure Aggregation}},
  booktitle = {ACSSC},
  year = {2019},
  doi = {10.1109/ieeeconf44664.2019.9049066},
}
Figure