Abstract
Federated Learning (FL) typically involves numerous communication rounds, incurring significant bandwidth and time costs. One-shot FL aims to produce a global model in a single round but often relies on compute-intensive Knowledge Distillation (KD) or proxy datasets. We present MA-Echo (Model Aggregation via Exploring Common Harmonized Optima), a novel one-shot FL algorithm that aggregates local models without server-side training or auxiliary data. By leveraging null space projection, MA-Echo identifies a common parameter space that preserves local knowledge, outperforming existing methods in extreme non-IID scenarios.
Introduction
The primary bottleneck in standard Federated Learning is the high communication overhead required for convergence. While one-shot FL offers a solution, current state-of-the-art methods frequently utilize Knowledge Distillation, which introduces privacy risks, high computational costs at the server, and dependencies on public datasets.
MA-Echo addresses these challenges by treating model aggregation as a forgetting-alleviation problem. It seeks to find a global model that minimizes loss across all clients simultaneously without requiring access to client data or performing heavy server-side optimization.
Methodology: Exploring Harmonized Optima
The core of MA-Echo is the identification of a "common harmonized optima" through several key technical innovations.
Null Space Projection
For any neural network layer, certain directions in the parameter space are orthogonal to the input features. Updates in these "null space" directions do not alter the model's output. MA-Echo utilizes this property to update parameters in a way that preserves the specific knowledge learned by each local client.
Multi-Objective Optimization and SVM Formulation
The server solves a multi-objective optimization problem to find a global model that satisfies all local constraints. The authors demonstrate that the dual of this objective can be efficiently solved as a One-class Support Vector Machine (SVM), enabling the use of standard, high-performance optimization libraries.
Layer-wise Aggregation
The algorithm operates layer-by-layer, ensuring compatibility with various architectures like MLPs and CNNs. It can be further enhanced with neuron-matching techniques, such as Optimal Transport, to align features before the projection step.
Experimental Evaluation
The researchers evaluated MA-Echo on standard benchmarks (MNIST, CIFAR-10/100) and complex heterogeneous datasets (FEMNIST, DomainNet).
Key Results
- Non-IID Performance: In extreme non-IID settings (e.g., MNIST with no overlapping labels), MA-Echo achieved 80.31% accuracy, significantly higher than FedAvg (20.97%) and the distillation-based DENSE (32.77%).
- Efficiency: MA-Echo is orders of magnitude faster than distillation methods. For a 50-client setup, it aggregated models in under 10 seconds, compared to over 20 minutes for DENSE.
- Generative Capabilities: When applied to Conditional Variational Auto-Encoders (CVAEs), the aggregated global model successfully generated images for all categories, even those not present in individual local datasets.
Discussion and Insights
MA-Echo shifts the perspective of the server from a "trainer" to a "harmonizer." By viewing aggregation through the lens of memory preservation, the method avoids the "catastrophic forgetting" often seen when simply averaging models trained on disjoint data. Furthermore, the use of Singular Value Decomposition (SVD) allows for compressing the required projection matrices by up to 800x with minimal performance loss.
Conclusion and Future Work
While MA-Echo provides a robust and efficient path for one-shot FL, it faces challenges when the rank of input feature subspaces is high, potentially shrinking the available null space. Future research will focus on dimensionality reduction techniques and extending the approach to more complex architectures like large-scale Transformers.
Summary updated