Mean Aggregator Is More Robust than Robust Aggregators under Label Poisoning Attacks

Jie Peng, Weiyu Li, Qing Ling

Introduction

Distributed learning, particularly Federated Learning (FL), relies on a central server to aggregate model updates from multiple workers. While this architecture enables large-scale training, it is inherently vulnerable to malicious participants. Traditional research has focused heavily on Byzantine attacks, where workers can send arbitrary, worst-case messages. To mitigate this, "robust aggregators" like Trimmed Mean, Median, or Centered Clipping were developed to filter out outliers.

However, this paper identifies a critical oversight: in the practical scenario of label poisoning attacks—where workers simply use mislabeled data—these robust aggregators are often counterproductive. The study demonstrates that the simple Mean aggregator is not only more robust but theoretically optimal in these settings, especially as data heterogeneity increases.

The Heterogeneity Trap

Robust aggregators operate on the principle of finding a consensus. They assume that malicious updates will appear as outliers compared to the "honest" majority. In an IID (Independent and Identically Distributed) setting, this assumption holds. However, real-world data is often highly heterogeneous (non-IID).

In non-IID environments, legitimate workers naturally produce diverse gradients. Robust aggregators often mistake this natural diversity for a malicious attack, discarding valuable information from honest workers. This "heterogeneity trap" leads to significant performance degradation, whereas the Mean aggregator preserves the collective knowledge of all honest participants.

Theoretical Optimality

The researchers provide a formal convergence analysis for both Mean and $\rho$-robust aggregators under label poisoning. They prove that for common tasks like softmax regression, the "disturbance" caused by poisoned labels is bounded.

Crucially, they show that the learning error of the Mean aggregator is $O(\delta^2 \xi^2)$, where $\delta$ represents the fraction of poisoned workers and $\xi$ denotes the degree of data heterogeneity. This result matches the theoretical lower bound, indicating that the Mean aggregator achieves the best possible order of magnitude for learning error under these conditions. Furthermore, while robust aggregators like TriMean and FABA experience "error explosions" as the poisoning ratio increases, the Mean aggregator remains remarkably stable.

Empirical Validation

The authors validated their theoretical findings using MNIST and CIFAR-10 benchmarks across varying degrees of data distribution: IID, mild non-IID (Dirichlet distribution), and extreme non-IID.

Accuracy comparison across different data distributions and datasets

As shown in Figure 5, the Mean aggregator (red line) consistently outperforms robust alternatives in non-IID settings. In the most challenging non-IID scenarios for both MNIST and CIFAR-10, the performance gap between the Mean and robust aggregators like FABA or TriMean becomes substantial.

The Optimal Regime for Mean Aggregation

To provide a practical guide for practitioners, the authors mapped the "best aggregator" across a spectrum of attack probabilities ($p$) and data heterogeneity levels ($\alpha$).

Heatmap of the best aggregator choice across varying levels of heterogeneity and attack intensity

Figure 6 illustrates that as data becomes more heterogeneous (lower $\alpha$) or the attack strength decreases, the Mean aggregator (orange) becomes the dominant choice. Robust aggregators (like LFighter, shown in light blue) only maintain an advantage in relatively homogeneous settings with high attack probabilities.

Conclusion

This research challenges the prevailing wisdom that robust aggregators are always superior in adversarial environments. For the common case of label poisoning, the Mean aggregator's ability to incorporate all available information—even if slightly noisy—outperforms the conservative filtering of robust methods. In the presence of high data heterogeneity, the Mean aggregator is the most effective and theoretically sound choice for distributed learning systems.

Limitations and Future Work

The current theoretical framework is based on a deterministic setting using full gradients. Future work is needed to extend these proofs to the stochastic setting (mini-batches) and to explore these dynamics in fully decentralized, peer-to-peer learning environments.

Summary updated

BibTeX
@inproceedings{peng2024mean,
  author = {Jie Peng and Weiyu Li and Qing Ling},
  title = {{Mean Aggregator Is More Robust than Robust Aggregators under Label Poisoning Attacks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure