Introduction
In Federated Learning (FL), a central server aggregates model updates from distributed clients to train a global model. However, this decentralized nature makes FL vulnerable to "Byzantine" attacks, where malicious actors send poisonous updates to degrade model performance or inject stealthy backdoors. Traditional defenses, such as Krum or Bulyan, attempt to identify these attackers by measuring Euclidean distances between client weight updates in high-dimensional parameter spaces.
This paper argues that parameter-based detection is fundamentally flawed for complex tasks like Natural Language Processing (NLP). In these settings, a malicious update can appear benign within the massive parameter space of a neural network while originating from a drastically different data distribution. To address this, the authors introduce Client Embedding, a method that maps clients into a low-dimensional space where distances represent actual data distribution divergences.
The Client Embedding Framework
The proposed method is a "plug-and-play" enhancement for existing Byzantine-resistant aggregators. The process involves four key steps:
- Modeling Divergence: The framework assumes that the distance between two client embeddings should reflect the $f$-divergence between their underlying data distributions.
- Estimation: Since the server cannot access raw client data, it uses a mathematical proxy—an $f$-divergence indicator derived from the difference in client parameters and an estimated diagonal Hessian matrix.
- Dimensionality Reduction: Using a technique based on Cholesky decomposition, the server solves for an $(n-1)$-dimensional embedding matrix that satisfies the calculated divergence constraints.
- Enhanced Aggregation: Standard algorithms (like Krum) are then executed using these embedding distances instead of raw parameter distances.
Experimental Results and Separability
The researchers evaluated the method using LSTM models on the SST-2 and Amazon Reviews datasets, testing against various attacks including Gaussian noise and stealthy backdoors (BadWord and BadSent).
A key insight is that while malicious and clean clients are often intermingled in the high-dimensional parameter space, they become clearly separable in the proposed embedding space. This visual separability translates directly into improved detection metrics. For instance, when using the Krum aggregator, the Mean Average Precision (MAP) for detecting malicious clients jumped from 21.9% to 58.9%.
Furthermore, the defense performance improved dramatically. Under a backdoor attack, the Attack Success Rate (ASR) for a Krum-based model dropped from a near-total failure of 99.95% to just 18.17% when enhanced with Client Embedding.
Conclusion and Future Work
Client Embedding significantly boosts the robustness of Federated Learning by shifting the focus from weight values to data distributions. While highly effective for LSTMs in NLP tasks, future work is needed to explore its efficacy on modern transformer architectures and Large Language Models (LLMs). Additionally, while the low-dimensional nature of the embeddings offers some inherent privacy benefits, formal differential privacy guarantees remain an area for further investigation.
Summary updated