Abstract
Standard Federated Learning (FL) assumes a global model or disjoint clusters, but real-world participants often belong to multiple overlapping groups. This paper introduces DP-OGL, a framework that provides the first formal privacy analysis of how sensitive information propagates and "fades" through these overlaps. By leveraging structured group learning, the authors demonstrate improved model accuracy while maintaining rigorous differential privacy guarantees.
Introduction: The Challenge of Overlapping Groups
In traditional FL, workers are typically treated as a single pool or partitioned into non-overlapping clusters. However, real-world entities like hospitals or mobile devices often form groups based on geography or trust policies that naturally overlap. For instance, a specialized clinic might participate in both a regional network and a national research consortium.
These overlaps create pathways for privacy leakage. When a worker in one group shares updates that are subsequently merged into another group by a common member, private data propagates across the network. Existing privacy accounting methods often fail to capture these multi-hop leaks, leading to overly pessimistic privacy bounds.
The DP-OGL Framework
To address these challenges, the authors propose DP-OGL (Differential Private Overlapping Grouped Learning) and its variant DP-OGL+.
Grouped Training and the $S$ Parameter
In DP-OGL, workers alternate between "intra-group" epochs (local training) and "inter-group" epochs (model merging via common workers). A critical component is the $S$ parameter, which controls the frequency of information mixing between groups. Increasing $S$ introduces a propagation delay, effectively confining sensitive information within its original group for longer periods.
DP-OGL+ for Enhanced Peer Privacy
For scenarios where workers trust immediate peers but not distant outsiders, DP-OGL+ employs a modified clipping and noise addition strategy. This approach hardens the system against distant adversaries by strategically scheduling noise injection.
Formal Privacy Analysis
The core contribution of this work is a novel Renyi Differential Privacy (RDP) analysis that quantifies two primary effects:
- Propagation Delay: The time required for information to traverse the group graph.
- Information Degradation: Similar to a "Whisper Game," model updates lose precision as they are repeatedly clipped and noised at each group master, causing the original private "signal" to weaken over distance.
Experimental Evaluation
The researchers evaluated DP-OGL using MNIST and Fashion-MNIST datasets with non-i.i.d. data distributions. They tested various topologies, including Ring and Label-based structures.
Key Results
- Utility Gains: DP-OGL consistently outperformed the global baseline (DP-FedAvg). In Ring structures, the grouped approach achieved significantly lower training loss, demonstrating that personalization via groups improves accuracy.
- Privacy Amplification: Theoretical bounds confirmed that privacy is "stronger" against distant workers. In some cases, leakage to a distant worker was 1.5x lower than to a direct peer.
- Effectiveness of DP-OGL+: Under the peer-trust model, DP-OGL+ reduced privacy leakage bounds by over 40% compared to standard DP-OGL.
Conclusion and Future Directions
This work establishes that privacy in overlapping groups is a function of distance and time. While current proofs for information degradation are focused on "string-like" topologies, future work aims to generalize these findings to complex web-like graphs and dynamic group memberships. Additionally, exploring Local DP variants could further reduce the need to trust central group masters.
Summary updated