Introduction
Distributed machine learning relies on a central server to aggregate updates from multiple workers. However, this architecture is inherently vulnerable to Byzantine attacks, where malicious nodes send "poisoned" gradients to derail model convergence. While robust aggregation rules (RARs) like Krum or Trimmed Mean exist, they often suffer from an "excessive caution" problem. By assuming all honest workers should look similar (the IID assumption), these methods frequently discard honest but unusual "outlier" gradients—a common occurrence in heterogeneous, non-IID Federated Learning environments.
This paper introduces $\mathcal{F} \circ LS$, a hybrid framework that resolves the tension between robustness and inclusion. Instead of a binary "keep or drop" approach, it uses Linear Scalarization (LS) to penalize suspected gradients based on their deviation from a reference norm, allowing the model to learn from diverse data without compromising security.
The $\mathcal{F} \circ LS$ Framework
The authors frame Byzantine robustness as a Multi-Objective Optimization (MOO) problem. The $\mathcal{F} \circ LS$ method operates in three primary steps:
- Partitioning: An existing robust aggregator ($\mathcal{F}$) is used to divide workers into two sets: "trusted" ($\mathcal{G}_t$) and "suspected" ($\mathcal{G}_b$).
- Gradient-Based Penalization: Rather than discarding the suspected set, each worker $i \in \mathcal{G}_b$ is assigned a weight $\lambda_i$. This penalty is calculated based on the gradient's distance from a reference update (typically the output of the robust aggregator).
- Linear Scalarization: The final model update is a weighted sum of all gradients. Trusted workers receive a standard weight, while suspected workers are included with a weight that decreases as their "strangeness" increases.
This "corrective" mechanism ensures that honest outliers—such as data from a specialized medical clinic in a broader healthcare network—are not silenced.
Key Results and Performance
The researchers evaluated $\mathcal{F} \circ LS$ across standard benchmarks (MNIST, CIFAR-10, SVHN) under various attack scenarios, including ALIE, Bit Flip, and Min-Max attacks.
A critical finding is that LS-enhanced versions of standard aggregators consistently outperform their vanilla counterparts, particularly when data is highly non-IID. For instance, on skewed MNIST data under a "Min-Sum" attack, the standard Coordinate-wise Median (CM) achieved only 12.8% accuracy, whereas the LS-enhanced version (CMLS) reached 89.0%.
As illustrated in the figure above, the LS variants (KLS, RFA_LS, trLS, CMLS) maintain higher accuracy throughout training compared to the base robust aggregators (Krum, RFA, trMean, CM) when facing the ALIE attack on CIFAR-10.
The Inclusion-Robustness Trade-off
The paper formalizes the idea that robustness should not come at the cost of marginalizing honest clients. By adjusting the penalty factor $\alpha_b$, practitioners can tune the system's sensitivity to outliers.
The experiments show that a balanced approach ($\alpha_b = 1$) often leads to superior performance compared to more aggressive filtering ($\alpha_b = 0.1$). This demonstrates that the "suspected" gradients often contain valuable information that accelerates convergence and improves final model quality, provided their influence is managed meticulously.
Conclusion
$\mathcal{F} \circ LS$ offers a significant step forward for robust federated learning. By treating Byzantine defense as a gradient-based penalty problem rather than a filtering problem, it enables models to be both secure against malicious actors and inclusive of diverse, honest data. Future work will explore more sophisticated trade-off vectors and scaling this approach to larger, more complex architectures.
Summary updated