Mixture of Message Passing Experts with Routing Entropy Regularization for Node Classification

Xuanze Chen, Jiajun Zhou, Yadong Li, Jinsong Chen, Shanqing Yu, Qi Xuan

TL;DR

The authors propose GNNMoE, a framework that applies the "Mixture of Experts" (MoE) architecture to Graph Neural Networks to solve the "one-size-fits-all" limitation of standard message passing. By using an entropy-driven routing mechanism, the model dynamically learns whether a specific node needs to aggregate information from its neighbors (homophily) or rely more on its own features (heterophily), consistently outperforming state-of-the-art methods across diverse graph types.

What Problem Does This Paper Address?

Most classical Graph Neural Networks (GNNs) like GCN or GAT operate on the homophily assumption: the idea that connected nodes are likely to share the same labels or features (e.g., friends in a social network often share interests). However, real-world graphs are often heterophilous, where connected nodes are different (e.g., a fraudster connecting to many legitimate accounts).

Standard GNNs use a uniform message-passing scheme across the entire graph. This is suboptimal because, as shown in Figure 1, a single graph can contain a mix of neighborhood contexts. Some nodes are in highly consistent clusters, while others sit at the intersection of diverse communities.

![Fig.

  1. Example of complex neighborhood context in graphs.](/figures/art_d06b8da477fd497b942726a127557ce6)

Existing attempts to fix this often rely on static rules or coarse-grained partitioning. The challenge is creating a model that can adapt its learning strategy at the individual node level without massive computational overhead.

Methods

The core of the paper is GNNMoE, which breaks down the GNN process into specialized "experts" and a smart "router."

  1. Message Passing Decomposition: The authors decompose message passing into two operations: Propagation (P) (smoothing features across neighbors) and Transformation (T) (applying non-linear weights to a node's own features).
  2. The Experts: GNNMoE uses four expert networks representing different combinations of these operations: PP, PT, TP, and TT.
  3. Hybrid Routing Architecture:
    • Soft Routing: Each node's features are passed through a routing layer that assigns weights to the four experts.
  4. Enhanced Feed-Forward Network (EFFN): After the MoE blocks, the model uses a hard routing layer to choose the best activation function (SwishGLU, GEGLU, or REGLU) to further refine the representation.
  5. Routing Entropy Regularization: This is the "secret sauce." Instead of manually picking how many experts to use (top-$k$), the authors introduce a regularization term based on entropy.
  6. High entropy results in "soft weighting" (using many experts), which is better for complex, heterophilous nodes.
  7. Low entropy results in "soft top-$k$" (focusing on one or two dominant experts), which is better for simple, homophilous nodes.

![Fig.

  1. Illustration of GNNMoE architectures.](/figures/art_72b28273e0954e348eed01f75e408f99)

Datasets

The model was evaluated on 12 benchmark datasets to test its versatility:

  • 6 Homophilous Graphs: Computers, Photo, Coauthor CS, Coauthor Physics, Facebook, and Ogbn-arxiv.
  • 6 Heterophilous Graphs: Actor, Squirrel-fix, Chameleon-fix, Tolokers, Roman-empire, and Penn94.

These datasets cover a wide range of domains, from citation networks to social media and Wikipedia page-links.

Key Insights and Ideas

  • Nodes have distinct "tastes": The authors empirically proved that high-homophily nodes prefer propagation-heavy schemes (PP, PT), while low-homophily nodes favor transformation (TT).
  • Graph-level preferences: Homophilous graphs generally perform better with "hard" routing (picking the single best expert), whereas heterophilous graphs need "soft" routing (blending multiple experts) to handle their inherent noise and diversity.
Observation 1: Preferences of nodes with varying degrees of homophily.
  • Entropy as a Bridge: By controlling the entropy of the routing weights, GNNMoE can automatically transition between these two strategies (soft vs. hard routing) during training, making it truly adaptive to the specific graph it is processing.
Observation 2: Preference for expert routing strategies across different graphs.

Key Results

GNNMoE demonstrated significant performance gains over 20 baseline models, including vanilla GNNs, specialized heterophilous GNNs, and Graph Transformers.

  • Accuracy: GNNMoE achieved the best performance on 11 out of 12 datasets.
  • Relative Gains: It showed improvements of up to 20.47% over vanilla GNNs and 12.06% over other specialized heterophilous models.
  • Efficiency: Despite the complexity of having multiple experts, GNNMoE is highly efficient. On large datasets like Ogbn-arxiv, it was 2 to 7 times faster than many Graph Transformer competitors and avoided the Out-of-Memory (OOM) issues that plague global attention mechanisms.
Table 1: Node classification results across 12 benchmarks.

Limitations and Future Work

While GNNMoE is a powerful step forward for node classification, the authors note a few areas for growth:

  • Complex Graph Tasks: The current work focuses on node classification; future research could extend this MoE approach to link prediction or graph-level classification.
  • Theoretical Bounds: While the entropy regularization works well in practice, deeper theoretical exploration of the bounds of entropy-driven routing could yield even more robust models.
  • Broader Graph Types: Testing the framework on more complex structures, such as hypergraphs or dynamic graphs, remains an open direction.

Summary updated

Abstract

Graph neural networks (GNNs) have achieved significant progress in graph-based learning tasks, yet their performance often deteriorates when facing heterophilous structures where connected nodes differ substantially in features and labels. To address this limitation, we propose GNNMoE, a novel entropy-driven mixture of message-passing experts framework that enables node-level adaptive representation learning. GNNMoE decomposes message passing into propagation and transformation operations and integrates them through multiple expert networks guided by a hybrid routing mechanism. And a routing entropy regularization dynamically adjusts soft weighting and soft top-$k$ routing, allowing GNNMoE to flexibly adapt to diverse neighborhood contexts. Extensive experiments on twelve benchmark datasets demonstrate that GNNMoE consistently outperforms SOTA node classification methods, while maintaining scalability and interpretability. This work provides a unified and principled approach for achieving fine-grained, personalized node representation learning.
BibTeX
@misc{chen2025mixture,
  author = {Xuanze Chen and Jiajun Zhou and Yadong Li and Jinsong Chen and Shanqing Yu and Qi Xuan},
  title = {{Mixture of Message Passing Experts with Routing Entropy Regularization for Node Classification}},
  year = {2025},
  eprint = {2502.08083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.08083},
}
Figure