When training machine learning models across distributed organizations—such as hospitals, banks, or edge devices—centralizing raw data is often impossible due to privacy regulations, intellectual property concerns, or massive bandwidth costs.
The core insight of federated learning is to flip the traditional paradigm: instead of moving data to a centralized model, we move the model to the data. Rather than sharing raw datasets, distributed clients train local models on their private data and share only the resulting model parameters (weights and biases) with a central coordinator. The coordinator then synthesizes these individual updates into a single, unified global model.
The Intuition Behind Weight Averaging
At first glance, averaging the weights of neural networks trained on different datasets sounds like it shouldn't work. Neural networks are highly non-linear, and their loss landscapes are rife with local minima, saddle points, and permutation symmetries. If you randomly initialize two identical networks and train them on different data, their weights will diverge into completely different regions of the parameter space. Averaging their weights directly would yield a model that performs worse than either individual network.
Federated parameter averaging works because of a crucial constraint: shared initialization.
Standard Training (No Shared Init):
Client A (Init 1) ───► Local Minimum A ┐
├─► Naive Average = Poor Performance
Client B (Init 2) ───► Local Minimum B ┘
Federated Averaging (Shared Init):
┌──► Client A Updates ──► Local Basin A ┐
Global Init (W_t) ────┤ ├─► Weighted Average = W_{t+1}
└──► Client B Updates ──► Local Basin B ┘
As outlined by Roman Rischke et al., the central server broadcasts a standardized model architecture and an initial set of weights to all participating clients. Because every client starts from the exact same point in the parameter space, their local optimization trajectories remain within the same local loss basin during their training iterations.
When the central server collects these weights, computing their average is mathematically akin to averaging the optimization steps taken by each client. This acts as a proxy for computing gradients over the combined global dataset, effectively finding a central point in the parameter space that satisfies the optimization objectives of all participants simultaneously. As demonstrated by Jessica Liu et al., this iterative aggregation loop allows decentralized models to approximate the performance of a centrally trained model without ever exposing raw user data.
Accounting for Data Imbalance
In a perfect world, every distributed client would possess the same amount of clean, well-distributed data. In reality, federated environments are highly non-IID (not identically and independently distributed). One hospital might have tens of thousands of patient records, while a smaller clinic might only have a dozen.
If the central server performed a simple arithmetic mean of the incoming parameters, a tiny client with noisy data would exert the same influence on the global model as a massive client with a highly representative dataset. This would cause the global model to drift and degrade.
To solve this, the standard aggregation mechanism uses a weighted average based on local sample sizes. As formalized in early frameworks like the one examined by Ang Li et al., each client's parameter update is scaled by the ratio of its local dataset size to the total number of samples across all active clients:
$$W_{t+1} = \sum_{k=1}^{K} \frac{n_k}{N} W_{t+1}^k$$
Where $n_k$ is the number of samples on client $k$, $N$ is the total samples across all clients, and $W_{t+1}^k$ represents the parameters received from client $k$.
This weighting ensures that the global model's optimization trajectory is dominated by the most statistically significant updates. Holger R Roth et al. demonstrated the real-world efficacy of this approach by deploying weighted federated averaging across international clinical institutions with highly heterogeneous data distributions, achieving robust cross-institutional generalizability. Similarly, Dong Yang et al. extended this weighted aggregation mechanism to semi-supervised learning environments, proving that it can successfully integrate unannotated data across diverse international cohorts.
Breaking the Uniformity Constraint: Heterogeneous Architectures
Traditional federated averaging relies on a strict assumption: every client must train the exact same model architecture. However, this assumption breaks down when deploying models across hardware with varying computational capabilities. A high-end server cluster and a low-power mobile device cannot comfortably train the same model.
Recent breakthroughs have shifted the focus toward heterogeneous parameter aggregation, allowing clients to train customized or scaled-down versions of a model while still contributing to a unified global backbone.
Subnetwork Reconstruction
When edge devices have limited memory or compute power, they can train pruned or truncated versions of the global model. But how do you average vectors of different dimensions?
Yongzhe Jia et al. proposed a reconstruction strategy where the central server maintains a full-sized global backbone. When a client returns a pruned, lower-dimensional parameter set, the server maps these parameters back into their corresponding slots in the full-sized architecture, filling in any missing values with the existing weights from the global backbone before performing the average.
Global Backbone: [ W1, W2, W3, W4, W5 ]
Client A (Pruned): [ W1, __, W3, __, W5 ]
Server Recovery: [ W1_A, W2_Global, W3_A, W4_Global, W5_A ] -> Ready for Averaging
Variable Width Aggregation
Instead of static pruning, Haozhe Su et al. introduced dynamic subnetwork scheduling. Devices train subnetworks of varying widths depending on real-time network conditions and hardware capacity. The central server tracks which parameters are shared across multiple subnetwork sizes and applies specialized averaging techniques to ensure that overlapping parameters are updated fairly, preventing narrower, frequently updated layers from biasing the wider, less frequently updated layers.
Subspace Alignment via SVD
For large language models (LLMs), clients often train Low-Rank Adaptation (LoRA) adapters to minimize compute overhead. However, different clients might require different adapter ranks based on their local task complexity.
To aggregate these structurally mismatched updates, xin tong et al. developed a method that mathematically decouples adapters into shared and personalized subspaces. When clients return updates with varying ranks, the central server applies Singular Value Decomposition (SVD) to compress, project, and align these heterogeneous subspaces into a coherent, uniform global update.
Broad Applications and Robustness
By decoupling model optimization from direct data access, parameter aggregation has unlocked collaborative machine learning across highly sensitive domains.
In healthcare, implementations like those described by Zahra Tabatabaei et al. and Zhijun Hu et al. use federated averaging to train diagnostic models across international boundaries, integrating multimodal data like CT, MRI, and PET scans without violating patient confidentiality.
The flexibility of the framework also extends to complex data structures. For instance, Sicong Che et al. demonstrated how weighted parameter averaging can be adapted for multi-view learning, combining disparate feature types across isolated populations to build generalized representations.
Ultimately, whether applied to clinical decision support systems (Chu Myaet Thwal et al.), mobile health monitoring (Xiaohang Xu et al.), or industrial time-series forecasting (Shaoming Zhu et al.), the core mechanism remains the same: leveraging shared initialization and decentralized optimization to extract collective intelligence from isolated data silos.
Key Foundational Work
Privacy-Preserving Multiparty Learning For Logistic Regression (2018) — contributed 1 concept elements to this direction.
PPLSA: Parallel Probabilistic Latent Semantic Analysis Based on MapReduce (2012) — contributed 1 concept elements to this direction.