Gain Without Pain: Offsetting DP-Injected Noises Stealthily in Cross-Device Federated Learning

Wenzhuo Yang, Yipeng Zhou, Miao Hu, Di Wu, Xi Zheng, Jessie Hui Wang, Song Guo, Chao Li

Introduction

Federated Learning (FL) enables collaborative model training across decentralized devices without sharing raw data. However, privacy remains a concern as sensitive information can still be leaked from shared gradients. Differential Privacy (DP) is the standard defense, injecting Gaussian noise into gradients to mask data. Unfortunately, this introduces a significant "utility-privacy trade-off," where the noise required for privacy degrades model accuracy by 10–30%, often making it impractical for high-precision applications.

This paper introduces Noise Information Secretly Sharing (NISS), a novel algorithm designed to achieve "Gain without Pain." NISS allows clients to inject high levels of local noise for privacy while mathematically canceling that noise during server-side aggregation, recovering the lost accuracy.

The NISS Algorithm

The core innovation of NISS is the strategic use of Secret Sharing (SS) to manage DP noise. Instead of treating noise as a permanent distortion, NISS makes it "disappearing" at the aggregate level.

Conceptual overview of noise sharing and offsetting between clients.

The process follows five key steps:

  1. Noise Generation: Each client $k$ generates Gaussian noise $n^k$ to protect their local gradient.
  2. Secret Sharing: The client splits a negated version of this noise into multiple shares and distributes them to "neighbor" clients.
  3. Distortion for Security: To prevent collusion between a malicious server and neighbors, clients apply a random distortion variable ($s_i$) to received shares. This ensures privacy even if some neighbors are untrustworthy.
  4. Uploading: Clients upload their gradient plus their own noise, plus the sum of negated noise shares received from others.
  5. Aggregation: When the Parameter Server (PS) sums the updates, the original noises and their negated counterparts offset each other. In a trustworthy environment, the noise cancels out perfectly, leaving a clean aggregate gradient.

Experimental Evaluation

The researchers evaluated NISS using MNIST and CIFAR-10 datasets across both IID and non-IID settings, comparing it against standard DP-FedAVG and the noise-free FedAVG baseline.

Test accuracy over training rounds for MNIST and CIFAR-10 under various settings.

Key Results

  • Accuracy Recovery: NISS improved model accuracy by an average of 21% compared to standard DP-FedAVG. In non-IID MNIST scenarios, the boost reached 30–40%.
  • Near-Baseline Performance: In scenarios with trustworthy clients, NISS achieves accuracy levels nearly identical to the noise-free FedAVG baseline.
  • Robustness to Attacks: Using the Deep Leakage from Gradients (DLG) metric, the authors demonstrated that NISS effectively blocks image reconstruction attacks that standard DP and FedAVG fail to prevent. Because noise cancels out during aggregation, clients can inject much larger local noise than standard DP allows, significantly hardening the defense.

Key Insights

  • The "Offsetting" Magic: By making noises negatively correlated across the network, NISS protects individual privacy (where noise is high) while ensuring model utility (where the sum of noise is zero).
  • Stealthy and Tunable Defense: The system uses a distortion factor $\tau$ to balance security and utility. In highly untrustworthy environments, $\tau$ can be increased to retain some noise in the final aggregate, providing a safety net at a minor cost to accuracy.

Limitations and Future Work

Despite its performance gains, NISS introduces peer-to-peer (P2P) communication overhead, which may be challenging for resource-constrained mobile devices. It also requires a "tracker server" to facilitate neighbor discovery. Future research will focus on optimizing NISS for large-scale IoT environments and modeling dynamic device trust to further mitigate collusion risks.

Summary updated

BibTeX
@article{yang2022gain,
  author = {Wenzhuo Yang and Yipeng Zhou and Miao Hu and Di Wu and Xi Zheng and Jessie Hui Wang and Song Guo and Chao Li},
  title = {{Gain Without Pain: Offsetting DP-Injected Noises Stealthily in Cross-Device Federated Learning}},
  journal = {IEEE Internet of Things Journal},
  year = {2022},
  url = {https://openreview.net/forum?id=tOiEOzXC6bY},
}
Figure