In distributed and federated learning, a central server typically aggregates local model updates from multiple clients using an arithmetic mean—a method widely known as Federated Averaging (FedAvg). While computationally efficient, the arithmetic mean possesses a breakdown point of zero. This means that a single arbitrary or malicious update can completely dictate the trajectory of the global model. In a Byzantine threat model, where compromised clients can submit poisoned gradients with extreme magnitudes or inverted directions, standard averaging inevitably leads to model collapse.
To build a secure system, the aggregation process must shift from linear averaging to non-linear, robust statistical estimation. The core intuition is simple: honest clients computing gradients on clean data will naturally produce updates that cluster together in Euclidean space because they are optimizing similar loss landscapes. Malicious clients attempting to derail the model must submit updates that deviate significantly from this consensus, rendering them geometric outliers.
The Geometric Median and High-Dimensional Centrality
The most direct way to replace the arithmetic mean is with a multi-dimensional centrality measure. While a standard median is easily computed for one-dimensional data, extending it to high-dimensional parameter spaces requires the geometric median. The geometric median of a set of vectors minimizes the sum of Euclidean distances to all points in the set.
As explored by Sham M. Kakade et al. in Robust Aggregation for Federated Learning, substituting the mean with the geometric median provides a high breakdown point of up to 50%. Why does this work? When an attacker scales a malicious gradient vector toward infinity, its influence on the geometric median's objective function is capped because the derivative of the Euclidean distance is a unit vector. Consequently, as long as more than half of the participating nodes are honest, the aggregate update remains firmly anchored within the consensus of the clean data. This property allows frameworks like the one proposed by Ali Jadbabaie et al. in Byzantine-Robust Federated Linear Bandits to maintain stable convergence and sublinear regret even under heavy adversarial presence.
Selection and Clustering Mechanisms
Rather than computing a global geometric center, some robust aggregation rules operate by evaluating the local neighborhoods of updates to select or isolate the most trustworthy contributions.
A foundational example of this selection-based strategy is Krum, introduced by Peva Blanchard et al. in Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent. Krum scores each submitted gradient by calculating the sum of distances to its closest neighbors. The update with the lowest score—meaning it sits in the densest cluster of vectors—is selected as the global update. Because honest updates outnumber malicious ones, the chosen gradient is guaranteed to point in a safe, representative direction.
Taking this geometric intuition further, the aggregation problem can be explicitly framed as a clustering problem with outliers. As demonstrated by Yuhao Yi et al. in Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers, the server can use 1-center or 1-mean objectives to find the smallest bounding ball that encompasses the majority of worker submissions. By stripping away the points that fall outside this dense cluster, the aggregator formally bounds the adversarial influence, ensuring near-optimal resilience against sophisticated, coordinated attacks.
Coordinate-Wise Trimming and Centered Clipping
An alternative to high-dimensional geometric operations is to apply robust estimation coordinate by coordinate. Methods like the coordinate-wise trimmed mean sort the values of each individual parameter across all clients and discard a fixed percentage of the highest and lowest values before averaging the remainder.
However, coordinate-wise methods can struggle when data is non-identically distributed (non-i.i.d.) across clients. In heterogeneous environments, an honest client might submit a gradient that looks like an outlier in certain dimensions simply because its local data distribution is unique. Discarding these updates leads to a severe loss of benign information.
To resolve this, Sai Praneeth Karimireddy et al. in Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing advocate for centered clipping. Instead of selecting a single vector or dropping coordinates entirely, centered clipping computes a central reference point and scales back any incoming update whose distance exceeds a specific threshold. This bounds the maximum damage a malicious actor can inflict without completely erasing the diverse, valuable features contributed by honest, heterogeneous clients. Similarly, Eduard Gorbunov in Secure Distributed Training at Scale utilizes iterative shifting mechanisms that pull individual gradient contributions toward a central consensus, neutralizing outliers while preserving the collective trajectory of the network.