ESMFL: Efficient and Secure Models for Federated Learning

Sheng Lin, Chenghong Wang, Hongjia Li, Jieren Deng, Yanzhi Wang, Caiwen DingarXiv 2020 · arXiv

Introduction

Federated Learning (FL) is a decentralized machine learning paradigm designed to preserve data privacy by training models locally on user devices. Despite its promise, FL faces two critical challenges: the communication bottleneck caused by massive model sizes and the privacy gap where gradient updates can leak sensitive information. ESMFL (Efficient and Secure Models for Federated Learning) addresses these issues by integrating hardware-based security with advanced model pruning.

The ESMFL Framework

The core of ESMFL is a dual-strategy approach that ensures both high security and extreme communication efficiency.

Hardware-Verified Security via Intel SGX

To protect against gradient leakage without the accuracy trade-offs of differential privacy, ESMFL utilizes Intel Software Guard Extensions (SGX). The central server, acting as a Cryptographic Service Provider (CSP), operates a Trusted Execution Environment (TEE) known as an Enclave.

ESMFL System Architecture: Integration of Enclave Space and Client-side pruning.

In this architecture, clients perform remote attestation to verify the enclave's integrity before transmitting encrypted model updates. Aggregation occurs within this hardware-protected "black box," preventing even server administrators from accessing raw gradients.

ADMM-Based Weight Pruning

To mitigate the communication bottleneck, ESMFL employs the Alternating Direction Method of Multipliers (ADMM) for model pruning. This process involves a "warm-up" phase with the full model followed by systematic regularization. Unlike simple magnitude pruning, ADMM treats pruning as a mathematical optimization problem, enabling significantly higher compression rates while maintaining model convergence and accuracy.

Experimental Evaluation

The framework was evaluated on MNIST (LeNet-5) and CIFAR-10 (MobileNetV2) datasets across both IID and Non-IID data distributions.

Key Results

ESMFL demonstrates that hardware-based security is highly compatible with model compression. Since SGX does not require adding noise to weights, models remain sparse, facilitating massive reductions in upload size.

  • Compression and Accuracy: On MNIST, ESMFL achieved compression rates between 9.99x and 87x with negligible accuracy loss (98.78% vs 98.26%).
  • Efficiency: For LeNet-5, the framework reduced communication costs by up to 48.78% and improved total elapsed time by 2.4x.
  • Robustness: The system maintains high performance even in challenging Non-IID scenarios.
Test accuracy over training rounds for Non-IID data at an 87x compression rate, comparing ESMFL to masked pruning.

On the more complex CIFAR-10 dataset using MobileNetV2, ESMFL reached 85.40% accuracy with a 4.95x compression rate, outperforming standard masked pruning baselines.

Discussion and Future Work

ESMFL successfully balances the "privacy vs. efficiency" trade-off. However, current limitations include hardware dependency on Intel SGX and increased computational overhead on the client side due to ADMM. Future research will explore support for other TEEs like ARM TrustZone and evaluate scalability with larger client populations.

Summary updated

Abstract

Nowadays, Deep Neural Networks are widely applied to various domains. However, massive data collection required for deep neural network reveals the potential privacy issues and also consumes large mounts of communication bandwidth. To address these problems, we propose a privacy-preserving method for the federated learning distributed system, operated on Intel Software Guard Extensions, a set of instructions that increase the security of application code and data. Meanwhile, the encrypted models make the transmission overhead larger. Hence, we reduce the commutation cost by sparsification and it can achieve reasonable accuracy with different model architectures.
BibTeX
@misc{lin2020esmfl,
  author = {Sheng Lin and Chenghong Wang and Hongjia Li and Jieren Deng and Yanzhi Wang and Caiwen Ding},
  title = {{ESMFL: Efficient and Secure Models for Federated Learning}},
  year = {2020},
  eprint = {2009.01867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.01867},
}
Figure