Joint Privacy Enhancement and Quantization in Federated Learning

Natalie Lang, Elad Sofer, Tomer Shaked, Nir Shlezinger

TL;DR

This paper introduces JoPEQ, a novel method for federated learning (FL) that simultaneously compresses model updates and enhances privacy. By cleverly leveraging the distortion introduced by vector quantization, JoPEQ achieves strong local differential privacy (LDP) guarantees and significant communication reduction without sacrificing model accuracy, outperforming existing methods in both privacy and efficiency.

What Problem Does This Paper Address?

Federated learning (FL) trains models across decentralized devices without sharing raw data. However, this paradigm faces two major hurdles: the substantial communication overhead from frequent model updates and the critical need to protect user privacy. Existing solutions often tackle these issues separately, leading to suboptimal trade-offs. Techniques like local differential privacy (LDP) add noise to protect privacy but increase data size, while compression methods reduce data size but can inadvertently expose more information. This paper addresses the challenge of jointly optimizing these two conflicting goals: reducing communication load and ensuring robust privacy in FL.

Methods

The core innovation of JoPEQ lies in its dual-function mechanism that integrates privacy enhancement and quantization. It builds upon Subtractive Dithered Quantization (SDQ), a universal vector quantization technique. The key idea is to harness the inherent distortion introduced by SDQ, which can be modeled as additive noise, to satisfy LDP requirements.

Here's a breakdown of the JoPEQ approach:

  • Dithered Quantization: JoPEQ uses dithered quantization, where a random "dither" signal is added to the model update before quantization. This dither is generated using a shared seed between the server and clients, allowing the server to remove its effect.
  • Privacy-Preserving Noise (PPN): Crucially, JoPEQ adds a low-power privacy-preserving noise (PPN) signal before quantization. This PPN is randomized using a local seed on each client, making it impossible for the server to perfectly reconstruct.
  • Joint Design: Instead of applying privacy mechanisms and compression separately, JoPEQ designs them jointly. The PPN is carefully chosen to work in tandem with the SDQ distortion. This allows JoPEQ to achieve desired LDP guarantees (specifically, multivariate t-distribution or Laplace mechanisms) while simultaneously compressing the model updates.
  • Universal Quantization: JoPEQ uses universal vector quantization based on random lattices, meaning it doesn't require prior knowledge of the data distribution, making it suitable for diverse FL settings.

Datasets

The paper evaluates JoPEQ on two standard datasets:

  • MNIST: Used for training and evaluating handwritten digit classification models with scalar quantizers (L=1).
  • CIFAR-10: Used for training and evaluating image classification models with multivariate quantizers (L=2).

Key Insights and Ideas

  • Leveraging Quantization Distortion for Privacy: The most significant insight is that the distortion inherent in quantization (specifically SDQ) can be effectively repurposed to satisfy LDP requirements. This avoids the need for separate, potentially more disruptive, privacy mechanisms.
  • Unified Approach Outperforms Disjoint Methods: JoPEQ consistently outperforms methods that apply compression and privacy enhancement separately. This is because the joint design allows for a more efficient allocation of the "noise budget" for privacy and the "distortion budget" for compression.
  • Privacy from Compression: In certain scenarios (high privacy budget $\epsilon$, specific quantizer choices), JoPEQ can achieve its privacy guarantees primarily through the compression-induced distortion, requiring minimal or no additional PPN.
  • Robustness to Model Inversion Attacks: JoPEQ demonstrates strong defense against model inversion attacks like Deep Leakage from Gradients (DLG) and its improved version (iDLG), effectively preventing the reconstruction of private data.

Key Results

  • Improved SNR: JoPEQ achieves significantly higher Signal-to-Noise Ratio (SNR) in received models compared to separate compression and privacy methods, especially in low bit-rate regimes. For example, with $\epsilon=3$ and $R=8$, JoPEQ shows a notable gain in SNR.
  • Competitive Accuracy: JoPEQ achieves accuracy comparable to vanilla FL (without privacy or compression) and other baselines, demonstrating that its privacy and compression mechanisms do not significantly degrade model utility. For instance, on MNIST, JoPEQ achieves similar test accuracies to FL and FL+SDQ across different model architectures.
  • Effective Privacy Guarantees: JoPEQ provides analytical LDP guarantees and is empirically shown to demolish model inversion attacks, making it difficult for adversaries to reconstruct training data.
  • Communication Efficiency: By compressing model updates, JoPEQ significantly reduces the amount of data transmitted, which is crucial for resource-constrained edge devices. For a CNN on CIFAR-10 with $R=1$, the bit-rate reduction is substantial compared to standard double-precision representations.
  • Convergence: Theoretical analysis shows that JoPEQ-aided FL converges at a rate comparable to standard FL, albeit with an additive term in the convergence bound that depends on the privacy level.

Limitations and Future Work

  • Shared Seed Dependency: JoPEQ relies on a shared seed for dither generation. The paper notes that if this seed is compromised or maliciously designed by the server, it could impact privacy. Future work could explore client-side seed generation.
  • Overloading Probability: The privacy guarantees are based on the assumption of vanishing overloading probability. In practice, some level of overloading might occur, and its impact on privacy needs further investigation.
  • External Adversaries: While JoPEQ is designed for LDP (protecting against untrusted servers), its effectiveness against external adversaries is not fully explored.
  • Fixed Bit-Rate: The current JoPEQ framework operates with a fixed bit-rate per iteration. Future work could explore adaptive bit-rate mechanisms.
  • Non-scaled Truncation: The paper suggests that applying non-scaled fixed truncation to model updates could address potential privacy leakage related to the scaling coefficient, but this is left for future investigation.
  • Alternative Privacy Mechanisms: The paper mentions that JoPEQ's framework can be extended to other privacy mechanisms beyond multivariate t-distribution and Laplace, such as $\ell_2$-mechanisms, which could be explored further.

Summary updated

Abstract

Federated learning (FL) is an emerging paradigm for training machine learning models using possibly private data available at edge devices. The distributed operation of FL gives rise to challenges that are not encountered in centralized machine learning, including the need to preserve the privacy of the local datasets, and the communication load due to the repeated exchange of updated models. These challenges are often tackled individually via techniques that induce some distortion on the updated models, e.g., local differential privacy (LDP) mechanisms and lossy compression. In this work we propose a method coined joint privacy enhancement and quantization (JoPEQ), which jointly implements lossy compression and privacy enhancement in FL settings. In particular, JoPEQ utilizes vector quantization based on random lattice, a universal compression technique whose byproduct distortion is statistically equivalent to additive noise. This distortion is leveraged to enhance privacy by augmenting the model updates with dedicated multivariate privacy preserving noise. We show that JoPEQ simultaneously quantizes data according to a required bit-rate while holding a desired privacy level, without notably affecting the utility of the learned model. This is shown via analytical LDP guarantees, distortion and convergence bounds derivation, and numerical studies. Finally, we empirically assert that JoPEQ demolishes common attacks known to exploit privacy leakage.
BibTeX
@article{lang2023joint,
  author = {Natalie Lang and Elad Sofer and Tomer Shaked and Nir Shlezinger},
  title = {{Joint Privacy Enhancement and Quantization in Federated Learning}},
  journal = {IEEE TSP},
  year = {2023},
  doi = {10.1109/tsp.2023.3244092},
}
Figure