One-shot Federated Learning without Server-side Training

Shangchao Su, Bin Li, Xiangyang Xue

Abstract

Federated Learning (FL) typically involves numerous communication rounds, incurring significant bandwidth and time costs. One-shot FL aims to produce a global model in a single round but often relies on compute-intensive Knowledge Distillation (KD) or proxy datasets. We present MA-Echo (Model Aggregation via Exploring Common Harmonized Optima), a novel one-shot FL algorithm that aggregates local models without server-side training or auxiliary data. By leveraging null space projection, MA-Echo identifies a common parameter space that preserves local knowledge, outperforming existing methods in extreme non-IID scenarios.

Introduction

The primary bottleneck in standard Federated Learning is the high communication overhead required for convergence. While one-shot FL offers a solution, current state-of-the-art methods frequently utilize Knowledge Distillation, which introduces privacy risks, high computational costs at the server, and dependencies on public datasets.

MA-Echo addresses these challenges by treating model aggregation as a forgetting-alleviation problem. It seeks to find a global model that minimizes loss across all clients simultaneously without requiring access to client data or performing heavy server-side optimization.

Methodology: Exploring Harmonized Optima

The core of MA-Echo is the identification of a "common harmonized optima" through several key technical innovations.

Conceptual illustration of model aggregation. (a) Loss surfaces for individual tasks and the global task. (b) Standard averaging. (c-d) The MA-Echo approach finding a harmonized optimum.

Null Space Projection

For any neural network layer, certain directions in the parameter space are orthogonal to the input features. Updates in these "null space" directions do not alter the model's output. MA-Echo utilizes this property to update parameters in a way that preserves the specific knowledge learned by each local client.

Multi-Objective Optimization and SVM Formulation

The server solves a multi-objective optimization problem to find a global model that satisfies all local constraints. The authors demonstrate that the dual of this objective can be efficiently solved as a One-class Support Vector Machine (SVM), enabling the use of standard, high-performance optimization libraries.

Layer-wise Aggregation

The algorithm operates layer-by-layer, ensuring compatibility with various architectures like MLPs and CNNs. It can be further enhanced with neuron-matching techniques, such as Optimal Transport, to align features before the projection step.

Experimental Evaluation

The researchers evaluated MA-Echo on standard benchmarks (MNIST, CIFAR-10/100) and complex heterogeneous datasets (FEMNIST, DomainNet).

Key Results

  • Non-IID Performance: In extreme non-IID settings (e.g., MNIST with no overlapping labels), MA-Echo achieved 80.31% accuracy, significantly higher than FedAvg (20.97%) and the distillation-based DENSE (32.77%).
  • Efficiency: MA-Echo is orders of magnitude faster than distillation methods. For a 50-client setup, it aggregated models in under 10 seconds, compared to over 20 minutes for DENSE.
  • Generative Capabilities: When applied to Conditional Variational Auto-Encoders (CVAEs), the aggregated global model successfully generated images for all categories, even those not present in individual local datasets.

Discussion and Insights

MA-Echo shifts the perspective of the server from a "trainer" to a "harmonizer." By viewing aggregation through the lens of memory preservation, the method avoids the "catastrophic forgetting" often seen when simply averaging models trained on disjoint data. Furthermore, the use of Singular Value Decomposition (SVD) allows for compressing the required projection matrices by up to 800x with minimal performance loss.

Conclusion and Future Work

While MA-Echo provides a robust and efficient path for one-shot FL, it faces challenges when the rank of input feature subspaces is high, potentially shrinking the available null space. Future research will focus on dimensionality reduction techniques and extending the approach to more complex architectures like large-scale Transformers.

Summary updated

BibTeX
@article{su2023one,
  author = {Shangchao Su and Bin Li and Xiangyang Xue},
  title = {{One-shot Federated Learning without Server-side Training}},
  journal = {Neural Networks},
  year = {2023},
  doi = {10.1016/j.neunet.2023.04.035},
}
Figure