Robust Distributed Gradient Aggregation Using Projections onto Gradient Manifolds

Kwang In KimAAAI 2024 · OpenReview

Abstract

We study the distributed gradient aggregation problem where individual clients contribute to learning a central model by sharing parameter gradients constructed from local losses. However, errors in some gradients, caused by low-quality data or adversaries, can degrade the learning process when naively combined. Existing robust gradient aggregation approaches assume that local data represent the global data-generating distribution, which may not always apply to heterogeneous (non-i.i.d.) client data. We propose a new algorithm that can robustly aggregate gradients from potentially heterogeneous clients. Our approach leverages the manifold structure inherent in heterogeneous client gradients and evaluates gradient anomaly degrees by projecting them onto this manifold. This algorithm is implemented as a simple and efficient method that accumulates random projections within the subspace defined by the nearest neighbors within a gradient cloud. Our experiments demonstrate consistent performance improvements over state-of-the-art robust aggregation algorithms.

Introduction

In distributed collaborative learning, such as Federated Learning, a central server aggregates model updates (gradients) from many remote clients. The standard approach is simple averaging. However, this is highly vulnerable: a single "Byzantine" client—whether a malicious actor or a device with poor-quality data—can send nonsensical gradients that derail the entire training process.

Existing robust aggregation methods usually assume that all clients' data come from the same distribution (i.e.i.d.). In this scenario, all "good" gradients should cluster together, making outliers easy to spot. But in the real world, client data is often heterogeneous (non-i.i.d.). For example, different hospitals might have data on different rare diseases; their gradients should look different. In these cases, traditional robust methods often mistake legitimate, diverse updates for attacks, or fail to catch subtle adversaries that hide within the diversity of the "gradient cloud."

Methodology: Projections onto Gradient Manifolds

The core hypothesis of this work is that while clean gradients from heterogeneous clients vary, they still lie on a low-dimensional gradient manifold.

Visualizing the gradient manifold: In heterogeneous settings, clean gradients (blue) form a manifold rather than a tight cluster. The proposed method identifies outliers (red) by their distance from this manifold.

Instead of measuring distance from a single global average, the proposed algorithm evaluates a gradient's "anomaly degree" by projecting it onto the local manifold structure. Since the exact shape of the manifold is unknown, the algorithm estimates it locally. For any given gradient, it finds its $H$ nearest neighbors in the gradient cloud and uses them to construct a local linear approximation (a tangent space). The anomaly degree is then defined as the squared distance between the original gradient and its projection onto this estimated tangent space.

To implement this efficiently, the server uses a Random Projection Initialization phase. Before main training begins, the server sends several sets of random model parameters to clients. Clients return gradients for these random points, allowing the server to build an initial "map" of the gradient manifold and assign reliability weights to each client. This approach is "plug-and-play"—it generates weights for each client that can be used with standard aggregation formulas.

Experimental Evaluation

The researchers evaluated their method across six diverse image classification benchmarks to test various levels of complexity and heterogeneity: CIFAR10, CIFAR100, TinyImageNet, FMNIST, Kuzushiji49, and EMNISTL. To simulate "affected" clients, they used a class-symmetric flip model, where labels are reassigned randomly, forcing the client to produce erroneous gradients.

Key Results

The proposed algorithm consistently outperformed state-of-the-art robust aggregators and the baseline uniform averaging across 42 different experimental cases.

Experimental results across six benchmarks. The proposed method (Ours) consistently outperforms existing robust aggregation techniques, particularly as the percentage of affected clients increases.
  • High Heterogeneity Performance: On TinyImageNet (a complex dataset with 200 classes), when 70% of the clients were providing bad data, the manifold method improved accuracy by 19.43% over uniform averaging.
  • Stability: While other robust methods performed significantly worse than simple averaging when data was highly heterogeneous (because they discarded legitimate diverse data), the manifold approach remained stable.
  • Robustness to Outlier Neighbors: A key insight is that even if the "neighborhood" used to estimate the manifold contains some outliers, the linear reconstruction is naturally resistant to them as long as the neighborhood size is much smaller than the gradient dimension.

Conclusion and Future Work

This manifold-based approach provides a robust solution for gradient aggregation without requiring prior knowledge of the number of attackers or access to a clean server-side dataset. However, challenges remain. Coordinated attacks, where multiple malicious clients send identical erroneous gradients to "create" a fake manifold, could potentially trick the algorithm. Additionally, the initial mapping phase adds a one-time communication cost. Future work will focus on deeper theoretical convergence analysis and addressing coordinated adversarial strategies.

Summary updated

Abstract

We study the distributed gradient aggregation problem where individual clients contribute to learning a central model by sharing parameter gradients constructed from local losses. However, errors in some gradients, caused by low-quality data or adversaries, can degrade the learning process when naively combined. Existing robust gradient aggregation approaches assume that local data represent the global data-generating distribution, which may not always apply to heterogeneous (non-i.i.d.) client data. We propose a new algorithm that can robustly aggregate gradients from potentially heterogeneous clients. Our approach leverages the manifold structure inherent in heterogeneous client gradients and evaluates gradient anomaly degrees by projecting them onto this manifold. This algorithm is implemented as a simple and efficient method that accumulates random projections within the subspace defined by the nearest neighbors within a gradient cloud. Our experiments demonstrate consistent performance improvements over state-of-the-art robust aggregation algorithms.
BibTeX
@inproceedings{kim2024robust,
  author = {Kwang In Kim},
  title = {{Robust Distributed Gradient Aggregation Using Projections onto Gradient Manifolds}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=GjgF8Ta9JT},
}

Loading related content...

Figure