Robust Softmax Aggregation on Blockchain based Federated Learning with Convergence Guarantee

Huiyu Wu, Diego Klabjan

TL;DR

Researchers have developed SABFL (Softmax Aggregation on Blockchain-based Federated Learning), a decentralized framework that replaces the vulnerable central server with a blockchain. By using a novel softmax-based weighting system to aggregate model updates, the system remains robust against malicious attacks and data heterogeneity while providing a mathematical guarantee of convergence.

What Problem Does This Paper Address?

Standard Federated Learning (FL) allows multiple parties to train a model together without sharing their raw data. However, it has two major "Achilles' heels":

  1. Centralization: It usually relies on a single central server. If that server fails or is hacked, the whole process collapses.
  2. Poisoning Attacks: Malicious participants can upload "poisoned" model updates to degrade the global model's accuracy.

While Blockchain-based FL (BCFL) has been proposed to solve the centralization issue, existing methods often struggle with data heterogeneity (where participants have very different amounts or types of data) and lack strong theoretical proof that the model will actually converge to an optimal solution. Furthermore, many BCFL schemes use "in-house" tokens for rewards that have no real-world value, disincentivizing honest participation.

Methods

The authors introduce SABFL, which stands out through three key components:

  • The Architecture: Instead of a central server, SABFL uses a Proof-of-Stake (PoS) mechanism on an existing blockchain (like Ethereum). It divides participants into three roles: Workers (who train), Validators (who verify), and Miners (who aggregate). Using an established blockchain ensures rewards have actual liquidity and value.
  • Softmax Aggregation: This is the "secret sauce." Instead of just averaging updates, the miner assigns weights to each worker’s update using a softmax function applied to their loss values. Validators use their own local data to estimate the "population loss" for each worker's model. If a worker provides a high-quality update (low loss), they get a higher weight; if they provide a poisoned update (high loss), their influence is mathematically "squashed."
  • Convergence Guarantee: The paper provides a rigorous mathematical proof (Theorem 1) showing that under certain convex conditions, this softmax aggregation is guaranteed to converge to the global minimum, even in the presence of noise and varying learning rates.

Datasets

The framework was evaluated across a diverse set of tasks to ensure the results weren't a fluke of one specific model type:

  • MNIST & FashionMNIST: Image classification using Convolutional Neural Networks (CNN).
  • Cover Type: Predicting forest cover types using Multi-Layer Perceptrons (MLP).
  • IMDB: Sentiment analysis using Recurrent Neural Networks (RNN).

The researchers specifically tested these against "Label Flipping" attacks, where malicious nodes intentionally mislabel data to confuse the model.

Key Insights and Ideas

  • The "Stake" Matters: By selecting validators based on their "stake" in the blockchain, the system assumes these high-stake players are more likely to have representative data and a vested interest in the system's integrity.
  • Sample Size is a Hidden Threat: One of the most interesting findings is that simply having participants with different sample sizes can be as damaging to traditional FL as having actual malicious attackers. SABFL handles this "size heterogeneity" much better than standard averaging.
  • Robustness to Collusion: The system remains stable even if up to half of the validators are malicious and collude with attackers. Because the aggregation is based on a consensus of loss values, a few bad actors can't easily tilt the scales.

Key Results

SABFL was tested in 16 different experimental settings (varying the number of participants, malicious nodes, and data distribution).

  • Dominance: Excluding "Vanilla" FL (which uses private sample size information that isn't always available), SABFL was the top performer in 13 out of 16 tasks.
  • Accuracy: In scenarios with 40% malicious participants, SABFL maintained high accuracy while traditional methods like Krum or Median aggregation saw significant performance drops.
  • Consistency: SABFL showed the lowest "ranking deviation," meaning it performs reliably well across different types of data and models, whereas other robust methods (like Krum) are "hit or miss" depending on the dataset.

Limitations and Future Work

While the results are impressive, there are a few areas for further exploration:

  • Convexity Assumptions: The convergence guarantee is currently proven for the convex setting. Most modern deep learning (like Large Language Models) is non-convex. While the experiments show it works well on neural networks, the theoretical proof for non-convexity is still an open challenge.
  • Validator Data: The system assumes validators have representative data. If all validators have extremely biased data, the approximation of the "population loss" might suffer.
  • Computational Overhead: Running these checks on a blockchain adds latency and cost (gas fees). Future work could look at optimizing the communication efficiency to make it more practical for high-frequency training.

Summary updated

BibTeX
@inproceedings{wu2024robust,
  author = {Huiyu Wu and Diego Klabjan},
  title = {{Robust Softmax Aggregation on Blockchain based Federated Learning with Convergence Guarantee}},
  booktitle = {COINS},
  year = {2024},
  doi = {10.1109/coins61597.2024.10622138},
}
Figure