BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning

Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang, Feng Yan, Yang Liu

TL;DR

This paper introduces BatchCrypt, a novel system that significantly speeds up cross-silo federated learning (FL) by efficiently batching and encrypting multiple gradient updates at once using homomorphic encryption (HE). This approach drastically reduces the computational and communication overhead of HE, making secure FL practical for real-world applications without sacrificing model accuracy.

What Problem Does This Paper Address?

Federated learning (FL) allows organizations to collaboratively train machine learning models without sharing their raw data, which is crucial for privacy-sensitive industries like finance and healthcare. Cross-silo FL, where participating entities are organizations rather than individual devices, often employs homomorphic encryption (HE) to ensure that even the aggregated model updates remain private from the central server. However, standard HE implementations are computationally expensive and generate very large encrypted data, leading to prohibitively long training times and massive communication costs. This overhead makes HE impractical for many real-world cross-silo FL deployments.

Methods

BatchCrypt tackles the HE overhead by introducing a novel "batch encryption" technique. Instead of encrypting each individual gradient update, BatchCrypt first quantizes gradient values into low-bit integers. Then, it encodes a batch of these quantized gradients into a single long integer, which is then encrypted with HE in one go. To enable gradient-wise aggregation on these encrypted batches, BatchCrypt also introduces:

  • Custom Quantization and Encoding Schemes: These are designed to preserve the additive properties of HE and handle signed integers effectively, using a two's complement representation with two sign bits.
  • Novel Gradient Clipping Technique (dACIQ): This analytical method determines optimal clipping thresholds for gradients in a distributed FL setting, minimizing quantization errors and preserving model quality.
  • Advance Scaling: This technique scales the quantization range based on the number of clients to proactively prevent overflows during aggregation.

BatchCrypt is implemented as a plug-in module for FATE, an existing cross-silo FL framework.

Datasets

The paper evaluates BatchCrypt using three standard deep learning models trained on publicly available datasets:

  • FMNIST: A dataset of Zalando's article images, used with a 3-layer fully-connected neural network.
  • CIFAR-10: A dataset of 60,000 32x32 color images in 10 classes, used with the AlexNet model.
  • Shakespeare: A dataset of Shakespeare's plays, used with an LSTM model for text generation.

The authors note that they partition these datasets across nine participating clients in a geo-distributed setup.

Key Insights and Ideas

  • HE is the Bottleneck: The paper's characterization clearly shows that HE operations (encryption/decryption) dominate training time (over 80%) and dramatically increase data transfer (over 150x) in cross-silo FL.
  • Batching is Key, but Tricky: While the idea of batching for HE efficiency has been mentioned, a practical, robust implementation for FL was missing. BatchCrypt provides this by co-designing quantization, encoding, and clipping.
  • Quantization is Necessary and Manageable: Quantization is essential for batching HE, and the paper demonstrates that with appropriate bit widths (e.g., 8-16 bits), it incurs negligible accuracy loss (<1%).
  • Gradient Clipping is Crucial for Quantization: Finding optimal clipping thresholds is vital to minimize quantization errors. The dACIQ method provides an analytical solution for this in a distributed FL context.
  • Model Placement Matters: Placing model weights on the client side, rather than the aggregator, allows BatchCrypt to use advanced optimizers like Adam on plaintext aggregated gradients, which is not possible with traditional HE setups.

Key Results

  • Significant Speedup: BatchCrypt achieves substantial training speedups: 23x-93x compared to the stock FATE implementation.
  • Reduced Communication: Communication overhead is reduced by 66x-101x.
  • Negligible Accuracy Loss: The accuracy loss due to quantization is less than 1%.
  • Cost Savings: BatchCrypt reduces network costs by up to 99% and overall training costs significantly, making cross-silo FL economically viable.
  • Scalability: The benefits of BatchCrypt are more pronounced for larger models.

Limitations and Future Work

  • Quantization Bit Width Sensitivity: While 8-bit quantization works well for FMNIST, it shows poorer performance for CIFAR and LSTM. The paper suggests that longer bit widths are generally better but acknowledges that finding the optimal width is model-dependent.
  • Vertical FL Applicability: Batching complex operations like ciphertext matrix multiplication in vertical FL is beyond the current scope of BatchCrypt and is left as future work.
  • Exploration of More Sophisticated Models: The paper suggests that more advanced ML models might be even more resilient to quantization noise, promising further improvements with BatchCrypt.
  • Flexible Synchronization: While BatchCrypt is evaluated in a synchronous setting, its design is compatible with flexible synchronization schemes, which could be explored further.

Summary updated

BibTeX
@inproceedings{zhang2020batchcrypt,
  author = {Chengliang Zhang and Suyi Li and Junzhe Xia and Wei Wang and Feng Yan and Yang Liu},
  title = {{BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning}},
  booktitle = {USENIX ATC},
  year = {2020},
}
Figure