One-shot Federated Learning without server-side training

Shangchao Su, Bin Li, Xiangyang Xue

TL;DR

MA-Echo is a novel one-shot federated learning algorithm that aggregates local models in a single communication round without requiring any server-side training or auxiliary public datasets. By leveraging "null space projection," it identifies a common optimal parameter space that preserves the unique knowledge of each client, significantly outperforming existing distillation-based and averaging methods in extreme non-IID (non-identical) data scenarios.

What Problem Does This Paper Address?

Federated Learning (FL) typically requires hundreds of communication rounds between a central server and distributed clients, which is prohibitively expensive in terms of bandwidth and time. "One-shot" FL aims to solve this by producing a high-quality global model in just a single round.

However, existing one-shot methods usually rely on Knowledge Distillation (KD). KD requires the server to either have access to a proxy public dataset or to spend significant compute power generating "pseudo-samples" to act as a bridge between models. This introduces new problems: data domain mismatches, high server-side computational costs, and potential privacy leaks. The authors of this paper ask: Can we aggregate models effectively without any data and without a heavy training phase on the server?

Methods

The authors propose MA-Echo (Model Aggregation via Exploring Common Harmonized Optima). The core innovation is treating model aggregation as a forgetting-alleviation problem, similar to techniques used in Continual Learning.

Key Techniques:

  • Null Space Projection: The intuition is that for any neural network layer, there are directions in the parameter space that are orthogonal to the input features. Moving parameters in these "null space" directions doesn't change the model's output. MA-Echo uses this to update the global model without "forgetting" what the local models learned.
  • Multi-Objective Optimization: Instead of a simple average, the server solves an optimization problem to find a global model that minimizes the loss across all local clients simultaneously.
  • One-class SVM Formulation: The authors cleverly show that the dual problem of their optimization objective can be solved as a One-class Support Vector Machine (SVM), allowing them to use efficient, standard optimization libraries.
  • Layer-wise Treatment: The algorithm processes the network layer-by-layer, making it compatible with both MLPs and CNNs. It can also be combined with "neuron-matching" techniques (like Optimal Transport) to align neurons before the projection step.

Datasets

The researchers evaluated MA-Echo across several standard and complex benchmarks:

  • MNIST & CIFAR-10: Used for baseline comparisons with MLPs and CNNs.
  • CIFAR-100: Used to test the aggregation of fine-tuned ResNet-18 models.
  • FEMNIST & DomainNet: Used to simulate real-world data heterogeneity and domain shifts (e.g., different styles like "clipart" vs. "real").
  • CVAE (Conditional Variational Auto-Encoders): Used for a fascinating qualitative test—aggregating decoders to see if the global model can generate images from categories it never saw in a single local dataset.

Key Insights and Ideas

  • Aggregation as Memory: The most profound shift here is viewing the server not as a "trainer," but as a "harmonizer" that tries to prevent the global model from destroying the specific knowledge learned by local clients.
  • Orthogonality is a Free Lunch: By identifying the "null space" of features, the server can find a consensus model that satisfies all clients without needing to see their raw data.
  • Efficiency over Distillation: Unlike methods like DENSE, which require training a generator and then performing distillation (taking hundreds of seconds), MA-Echo is an iterative parameter update that finishes in seconds.
  • Compression via SVD: The projection matrices required for the algorithm can be massive, but the authors found they could compress them by up to 800x using Singular Value Decomposition (SVD) while retaining 94% of the performance.

Key Results

  • Superior Accuracy in Extreme Non-IID Settings: In a 5-client MNIST setup where each client had zero overlapping labels with others ($\beta=0.01$), MA-Echo achieved 80.31% accuracy, compared to just 20.97% for standard averaging (FedAvg) and 32.77% for the distillation-based DENSE.
  • Speed: On a 50-client setup, MA-Echo took roughly 9.6 seconds to aggregate, while DENSE took over 1,270 seconds.
  • Generative Success: When aggregating CVAEs, the MA-Echo global model successfully generated all ten digits (0-9), even though each local model only knew a subset (e.g., one model knew {0,1,3,4,7} and the other {2,5,6,8,9}). Simple averaging failed to produce legible digits.

Limitations and Future Work

  • The Rank Problem: If the rank of the input feature subspace is too high (approaching the dimensionality of the parameters), the "null space" becomes too small, making it difficult to find an orthogonal direction for updates. The authors suggest dimensionality reduction as a potential fix.
  • Communication Overhead: While SVD helps, the requirement to send projection matrices still adds more communication overhead than simple FedAvg.
  • Model Complexity: While tested on ResNet-18, the authors note that future work is needed to refine the approach for even more complex, modern architectures like large-scale Transformers.

Summary updated

Abstract

Federated Learning (FL) has recently made significant progress as a new machine learning paradigm for privacy protection. Due to the high communication cost of traditional FL, one-shot federated learning is gaining popularity as a way to reduce communication cost between clients and the server. Most of the existing one-shot FL methods are based on Knowledge Distillation; however, {distillation based approach requires an extra training phase and depends on publicly available data sets or generated pseudo samples.} In this work, we consider a novel and challenging cross-silo setting: performing a single round of parameter aggregation on the local models without server-side training. In this setting, we propose an effective algorithm for Model Aggregation via Exploring Common Harmonized Optima (MA-Echo), which iteratively updates the parameters of all local models to bring them close to a common low-loss area on the loss surface, without harming performance on their own data sets at the same time. Compared to the existing methods, MA-Echo can work well even in extremely non-identical data distribution settings where the support categories of each local model have no overlapped labels with those of the others. We conduct extensive experiments on two popular image classification data sets to compare the proposed method with existing methods and demonstrate the effectiveness of MA-Echo, which clearly outperforms the state-of-the-arts. The source code can be accessed in \url{https://github.com/FudanVI/MAEcho}.
BibTeX
@article{su2023one,
  author = {Shangchao Su and Bin Li and Xiangyang Xue},
  title = {{One-shot Federated Learning without server-side training}},
  journal = {Neural Networks},
  year = {2023},
  doi = {10.1016/j.neunet.2023.04.035},
}
Figure