When MoE Meets Blockchain: A Trustworthy Distributed Framework of Large Models

Weihao Zhu, Long Shi, Kang Wei, Zhen Mei, Zhe Wang, Jiaheng Wang, Jun LiIEEE Communications Magazine 2025 · DOI · OpenReview

TL;DR

This paper introduces B-MoE, a novel framework that integrates Mixture of Experts (MoE) models with blockchain technology to enhance trustworthiness in distributed edge environments. By leveraging blockchain for verification and decentralized record-keeping, B-MoE significantly improves robustness against data manipulation attacks compared to traditional distributed MoE.

What Problem Does This Paper Address?

Large Models (LMs) are increasingly adopting the Mixture of Experts (MoE) architecture due to its efficiency in scaling model parameters without a proportional increase in computational cost. MoE works by using a gating network to selectively activate a subset of specialized "expert" neural networks for each task. While deploying MoE on distributed edge networks offers benefits like reduced latency and improved privacy, a critical challenge emerges: the lack of trust in data interactions among these distributed experts. Without a central authority, malicious actors (edges) can manipulate expert computations or model parameters, leading to severe performance degradation during both training and inference. Traditional MoE's gating network can mitigate some training-time attacks by reducing the activation of compromised experts, but this comes at the cost of inefficient resource utilization and still degrades performance. Crucially, during inference, the gating network cannot detect these manipulations, leading to significant accuracy drops. This paper addresses this security vulnerability by proposing a blockchain-integrated solution.

Methods

The proposed B-MoE (Blockchain-aided Trustworthy MoE) framework introduces a three-layer architecture:

  1. Edge Layer: This layer comprises distributed edge devices that download experts from a storage layer to perform computations. A key aspect is the redundancy mechanism: each edge downloads all activated experts for a given task. This allows for cross-verification of results.
  2. Blockchain Layer: This layer acts as the trust anchor. It hosts the gating network, receives computational results from the edge layer, and performs distributed consensus to verify and agree upon the trustworthy results. This layer uses a blockchain network to trace, verify, and record these results in a decentralized and immutable manner.
  3. Storage Layer: This layer utilizes a decentralized storage network (like IPFS) to store experts. Experts are identified by Content Identifiers (CIDs) stored on the blockchain, enabling efficient retrieval.

The workflow involves:

  • Task Publishing: Tasks are published on the blockchain.
  • Gate Evaluation: The on-chain gating network evaluates expert weights.
  • Expert Computation: Edges download tasks and all activated experts, compute results, and upload them to the blockchain.
  • Distributed Consensus: Blockchain nodes verify expert results by comparing outputs from different edges for the same expert. The majority-consistent results are deemed trustworthy.
  • MoE Updating (Training only): Edges update experts, and the blockchain updates the gating network based on feedback. Updated experts are stored, and their hashes are recorded on the blockchain.
  • Block Generation: Verified results and expert information are packaged into blocks.

The core novelty lies in using the blockchain for decentralized verification and consensus on expert computations, combined with the redundancy mechanism at the edge layer to enable this verification.

Datasets

The paper evaluates B-MoE on two standard image classification datasets:

  • Fashion-MNIST: A dataset of 28x28 grayscale images of clothing items.
  • CIFAR-10: A dataset of 32x32 RGB color images across 10 classes (e.g., aircraft, cats, dogs).

The experiments simulate data manipulation attacks by injecting random Gaussian noise into experts deployed on malicious edges, with a probability of 0.2 for each edge.

Key Insights and Ideas

  • Blockchain as a Trust Anchor: The paper convincingly demonstrates that integrating blockchain into distributed MoE can effectively address the trust deficit. The immutability and transparency of blockchain are crucial for ensuring the integrity of computational results.
  • Redundancy is Key for Verification: The requirement for each edge to download all activated experts, not just the top-K, is a critical design choice. This redundancy enables honest edges to compare results and identify malicious outputs, a capability absent in standard distributed MoE.
  • Training vs. Inference Vulnerability: The study highlights a significant difference in how MoE is vulnerable during training and inference. While the gating network can adapt to some attacks during training, it's largely ineffective against manipulated expert outputs during inference, making the blockchain's role even more critical for inference security.
  • The 50% Malicious Ratio Threshold: A crucial finding is that B-MoE's blockchain consensus mechanism can tolerate up to a 50% malicious ratio of edges. Beyond this threshold, the blockchain can be misled by the majority of malicious results, degrading performance. This is a fundamental limitation of many consensus mechanisms.

Key Results

  • Training Performance: Under data manipulation attacks, B-MoE achieved significant test accuracy improvements: at least 45% on Fashion-MNIST and 67% on CIFAR-10 compared to traditional distributed MoE. B-MoE's accuracy was comparable to non-attacked MoE.
  • Inference Performance: For malicious ratios below 50%, B-MoE showed improved inference accuracy: 66% on Fashion-MNIST and 44% on CIFAR-10 compared to traditional distributed MoE.
  • Latency Trade-off: B-MoE incurs higher latency compared to traditional distributed MoE, primarily due to the off-chain data interactions and on-chain consensus processes. However, this is presented as a trade-off for enhanced security.

Limitations and Future Work

  • Latency: The most significant limitation acknowledged is the increased latency introduced by the blockchain integration. Future work aims to optimize this through techniques like model compression (quantization, sparsification) and lightweight consensus mechanisms.
  • Malicious Ratio Threshold: The 50% malicious ratio threshold for blockchain consensus is a fundamental limitation. Future research could explore incentive mechanisms to encourage honest behavior and reduce the likelihood of reaching this threshold.
  • Resource Optimization: Deploying B-MoE on resource-constrained edge devices presents challenges. Future directions include developing resource optimization strategies, potentially using Lyapunov optimization and reinforcement learning.
  • Workload Balance: Ensuring balanced workload distribution among experts and edges in the B-MoE framework is an open problem. Dynamic expert scheduling and control mechanisms are suggested.
  • Incentive Mechanisms: Designing effective incentive mechanisms to encourage honest participation from edge nodes and mitigate risks associated with malicious behavior is a key area for future research.

Summary updated

Abstract

As an enabling architecture of Large Models (LMs), Mixture of Experts (MoE) has become prevalent thanks to its sparsely-gated mechanism, which lowers computational overhead while maintaining learning performance comparable to dense LMs. The essence of MoE lies in utilizing a group of neural networks (called experts) with each specializing in different types of tasks, along with a trainable gating network that selectively activates a subset of these experts to handle specific tasks. Traditional cloud-based MoE encounters challenges such as prolonged response latency, high bandwidth consumption, and data privacy leakage. To address these issues, researchers have proposed to deploy MoE over distributed edge networks. However, a key concern of distributed MoE frameworks is the lack of trust in data interactions among distributed experts without the surveillance of any trusted authority, and thereby prone to potential attacks such as data manipulation. In response to the security issues of traditional distributed MoE, we propose a blockchain-aided trustworthy MoE (B-MoE) framework that consists of three layers: the edge layer, the blockchain layer, and the storage layer. In this framework, the edge layer employs the activated experts downloaded from the storage layer to process the learning tasks, while the block-chain layer functions as a decentralized trustworthy network to trace, verify, and record the computational results of the experts from the edge layer. The experimental results demonstrate that B-MoE is more robust to data manipulation attacks than traditional distributed MoE during both the training and inference processes.
BibTeX
@misc{zhu2025when,
  author = {Weihao Zhu and Long Shi and Kang Wei and Zhen Mei and Zhe Wang and Jiaheng Wang and Jun Li},
  title = {{When MoE Meets Blockchain: A Trustworthy Distributed Framework of Large Models}},
  year = {2025},
  eprint = {2509.12141},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.12141},
}
Figure