Controlled privacy leakage propagation throughout overlapping grouped learning

Shahrzad Kiani, Franziska Boenisch, Stark C. Draper

Abstract

Standard Federated Learning (FL) assumes a global model or disjoint clusters, but real-world participants often belong to multiple overlapping groups. This paper introduces DP-OGL, a framework that provides the first formal privacy analysis of how sensitive information propagates and "fades" through these overlaps. By leveraging structured group learning, the authors demonstrate improved model accuracy while maintaining rigorous differential privacy guarantees.

Introduction: The Challenge of Overlapping Groups

In traditional FL, workers are typically treated as a single pool or partitioned into non-overlapping clusters. However, real-world entities like hospitals or mobile devices often form groups based on geography or trust policies that naturally overlap. For instance, a specialized clinic might participate in both a regional network and a national research consortium.

These overlaps create pathways for privacy leakage. When a worker in one group shares updates that are subsequently merged into another group by a common member, private data propagates across the network. Existing privacy accounting methods often fail to capture these multi-hop leaks, leading to overly pessimistic privacy bounds.

Comparison between standard FL (a) and the proposed overlapping group structure (b).

The DP-OGL Framework

To address these challenges, the authors propose DP-OGL (Differential Private Overlapping Grouped Learning) and its variant DP-OGL+.

Grouped Training and the $S$ Parameter

In DP-OGL, workers alternate between "intra-group" epochs (local training) and "inter-group" epochs (model merging via common workers). A critical component is the $S$ parameter, which controls the frequency of information mixing between groups. Increasing $S$ introduces a propagation delay, effectively confining sensitive information within its original group for longer periods.

DP-OGL+ for Enhanced Peer Privacy

For scenarios where workers trust immediate peers but not distant outsiders, DP-OGL+ employs a modified clipping and noise addition strategy. This approach hardens the system against distant adversaries by strategically scheduling noise injection.

Formal Privacy Analysis

The core contribution of this work is a novel Renyi Differential Privacy (RDP) analysis that quantifies two primary effects:

  1. Propagation Delay: The time required for information to traverse the group graph.
  2. Information Degradation: Similar to a "Whisper Game," model updates lose precision as they are repeatedly clipped and noised at each group master, causing the original private "signal" to weaken over distance.
Algorithm for Privacy-with-Propagation (PwP) Analysis.

Experimental Evaluation

The researchers evaluated DP-OGL using MNIST and Fashion-MNIST datasets with non-i.i.d. data distributions. They tested various topologies, including Ring and Label-based structures.

Key Results

  • Utility Gains: DP-OGL consistently outperformed the global baseline (DP-FedAvg). In Ring structures, the grouped approach achieved significantly lower training loss, demonstrating that personalization via groups improves accuracy.
  • Privacy Amplification: Theoretical bounds confirmed that privacy is "stronger" against distant workers. In some cases, leakage to a distant worker was 1.5x lower than to a direct peer.
  • Effectiveness of DP-OGL+: Under the peer-trust model, DP-OGL+ reduced privacy leakage bounds by over 40% compared to standard DP-OGL.

Conclusion and Future Directions

This work establishes that privacy in overlapping groups is a function of distance and time. While current proofs for information degradation are focused on "string-like" topologies, future work aims to generalize these findings to complex web-like graphs and dynamic group memberships. Additionally, exploring Local DP variants could further reduce the need to trust central group masters.

Summary updated

Abstract

Federated Learning (FL) is the standard protocol for collaborative learning. In FL, multiple workers jointly train a shared model. They exchange model updates calculated on their data, while keeping the raw data itself local. Since workers naturally form groups based on common interests and privacy policies, we are motivated to extend standard FL to reflect a setting with multiple, potentially overlapping groups. In this setup where workers can belong and contribute to more than one group at a time, complexities arise in understanding privacy leakage and in adhering to privacy policies. To address the challenges, we propose differential private overlapping grouped learning (DPOGL), a novel method to implement privacy guarantees within overlapping groups. Under the honest-but-curious threat model, we derive novel privacy guarantees between arbitrary pairs of workers. These privacy guarantees describe and quantify two key effects of privacy leakage in DP-OGL: propagation delay, i.e., the fact that information from one group will leak to other groups only with temporal offset through the common workers and information degradation, i.e., the fact that noise addition over model updates limits information leakage between workers. Our experiments show that applying DP-OGL enhances utility while maintaining strong privacy compared to standard FL setups.
BibTeX
@article{kiani2024controlled,
  author = {Shahrzad Kiani and Franziska Boenisch and Stark C. Draper},
  title = {{Controlled privacy leakage propagation throughout overlapping grouped learning}},
  journal = {IEEE J. Sel. Areas Inf. Theory},
  year = {2024},
}
Figure