Confederated Learning: Federated Learning With Decentralized Edge Servers

Xiaojun Yuan, Bin Wang, Jun Fang, Hongbin Li, Xiaojun Yuan 0002, Qing LingIEEE TSP 2023 · DOI

TL;DR

This paper introduces Confederated Learning (CFL), a novel multi-server federated learning framework designed to overcome the scalability limitations of traditional centralized FL. They propose an efficient ADMM-based algorithm that uses random user selection and inexact subproblem solving to significantly reduce communication overhead and accelerate convergence.

What Problem Does This Paper Address?

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models on decentralized data without compromising user privacy. However, traditional FL relies on a single central server to coordinate training across numerous devices. This centralized architecture faces significant scalability challenges as the number of devices grows, leading to communication bottlenecks and increased latency. While decentralized FL approaches exist, they often assume a fully peer-to-peer network, which may not be practical for current wireless edge networks. This paper addresses the need for a more scalable and efficient FL framework that can operate in a decentralized multi-server environment, specifically targeting the communication overhead inherent in such systems.

Methods

The core contribution is the Confederated Learning (CFL) framework, which features multiple edge servers (ESs) organized in a decentralized network. Each ES manages its own set of edge devices (users). To tackle the communication bottleneck, the authors propose an ADMM-based algorithm with two key innovations:

  • Random User Selection: At each iteration, only a random subset of users is activated to communicate with their respective ES. This significantly reduces the amount of data uploaded to the ESs.
  • Inexact Subproblem Solving: The algorithm allows the local subproblems (user model updates) to be solved up to a certain accuracy (ε-accuracy) rather than requiring exact solutions. This makes the process computationally less demanding for users.
  • Decentralized ES Communication: The algorithm is designed to allow ESs to communicate with their neighbors, enabling decentralized coordination without a global coordinator.
  • Extra Proximal Term: A carefully designed proximal term is added to the ES subproblem to facilitate its decentralized solution.

Datasets

The paper uses the Credit 1 dataset for evaluating their proposed method on an ℓ₂-regularized logistic regression problem. This dataset consists of 30,000 real data samples, each with 24 features and a binary label. For their experiments, they used 20,000 samples for training and assigned 20 samples per user.

Key Insights and Ideas

  • Decentralization is Key for Scalability: Moving beyond a single central server to a federated structure of multiple cooperating servers is crucial for scaling FL to massive edge networks.
  • Randomness as a Communication Saver: Randomly selecting users for participation in each round is a surprisingly effective strategy to drastically cut down on communication, a major bottleneck in FL.
  • Balancing Computation and Communication: The proposed method trades increased local computation (solving subproblems inexactly) for significantly reduced communication, a favorable trade-off given the increasing computational power of edge devices.
  • ADMM's Adaptability: The Alternating Direction Method of Multipliers (ADMM) proves to be a flexible framework that can be adapted to handle the complexities of decentralized multi-server FL.

Key Results

The paper demonstrates that their proposed CFL-ADMM algorithm significantly outperforms existing methods like GT-SAGA and D-SGD in terms of communication efficiency.

  • Faster Convergence: CFL-ADMM converges to a desired solution with substantially fewer communication rounds compared to other methods.
  • Impact of Parameters:
    • The error tolerance (ε) for subproblem solving affects the final accuracy, with smaller ε leading to solutions closer to the true optimum.
  • The user selection probability (α) impacts convergence speed, with values around 0.3 offering a good balance between performance and communication overhead.
  • Theoretical Guarantees: The authors provide theoretical analysis showing that the proposed algorithm achieves a sublinear convergence rate.

Limitations and Future Work

  • Computational Cost at Users: The CFL-ADMM algorithm requires users to solve subproblems up to a certain accuracy, which can involve multiple gradient descent iterations. While this is offset by the increased computational power of modern devices, it's a notable difference from methods that only require a single gradient computation per user.
  • Inexact Solution Accuracy: The paper acknowledges that when using a non-zero ε, the algorithm converges to a neighborhood of the optimal solution, not the exact optimum. Future work could explore strategies to further refine this accuracy or adapt ε dynamically.
  • Network Topology: The paper assumes a decentralized network of ESs but doesn't delve deeply into the impact of various network topologies on performance. Future research could investigate how different ES communication graph structures affect convergence.

Summary updated

BibTeX
@article{yuan2023confederated,
  author = {Xiaojun Yuan and Bin Wang and Jun Fang and Hongbin Li and Xiaojun Yuan and Qing Ling},
  title = {{Confederated Learning: Federated Learning With Decentralized Edge Servers}},
  journal = {IEEE TSP},
  year = {2023},
  doi = {10.1109/tsp.2023.3241768},
}
Figure