Letting Graph Filters Specialize by Competition

Graph neural networks (GNNs) compute each node's representation by repeatedly averaging it with its neighbors. That averaging acts as a low-pass filter. It smooths away differences between linked nodes, which helps when neighbors tend to share a label (homophily) and hurts when they tend not to (heterophily). The usual cure is to give the model a second, high-pass channel or to let each node attend over a fixed menu of filters. Either way, a human decides in advance what the filters are, and in the simplest versions one setting is shared by every node in the graph. Real graphs mix regimes: dense clusters where smoothing is right sit next to boundary regions where it erases exactly the signal that matters. One shared filter forces a compromise between them.

We propose to borrow a mechanism from mixture-of-experts learning, where parallel sub-networks ("experts") divide up the data among themselves without supervision. Give a GNN layer several learnable spectral filters that all start identical. On each labeled node, only the filter that makes the lowest loss is updated, a winner-takes-all rule. A small gate learns to predict that winner from a node's local features. The hypothesis is that this competition makes the filters split on their own into smoothing and sharpening experts, assigned node by node. It should also beat both hand-designed filterbanks and the simpler option of routing nodes by an estimate of their local homophily.

Core Claim

The claim is narrower than "mixture-of-experts fixes over-smoothing," and that is deliberate. Depth-induced collapse already has cheap fixes. The Oversmoothing Fallacy trains GCNs and graph attention networks to 1,024 layers with residual connections and batch normalization. Old can be Gold keeps a 64-layer vanilla GCN near 80% on Cora using isometric initialization and dynamic skip connections. We therefore do not pitch experts as a depth fix, because a reviewer would rightly answer "just add residuals."

Our claim concerns a different failure: frequency gradient interference in a filter shared across nodes. When some nodes need low-pass and others need high-pass filtering, their gradients on the shared filter coefficients point in opposite directions. The learned filter then lands at a compromise that serves neither group. This is the spectral analogue of the "instance gradient interference" that MAMMOTH found in pathology. In that work, tumor and stroma patches pull a shared linear layer in conflicting directions, and the conflict is resolved by soft-routed mini-experts.

We predict two things. First, the size of this conflict, measured before any experts are added, predicts on which datasets competitive filter experts help. Second, the competition rule, rather than the extra parameters, is what produces the gain.

Position Relative to Prior Work

Three existing graph mixture-of-experts methods place their experts outside message passing. GraphDIVE routes pooled whole-graph embeddings to linear heads to counter majority-class bias. Routing-by-Memory distills a GNN into a graphless mixture-of-experts student and explicitly leaves in-layer routing open. GraphMETRO runs parallel full GNN encoders, each targeting an externally injected corruption. None of them lets experts differ in how they filter over the graph, which is what we do.

The closest relatives work on the frequency side. FB-GNN splits the identity into fixed low-pass and high-pass operators and mixes them with two dataset-wide scalars. GSAN lets each node attend over GCN aggregation and precomputed scattering wavelets. Both fix the filter shapes in advance. Neither uses competition, so neither can test whether a low-/high-pass split is something optimization discovers. Ordered GNN partitions embedding neurons by hop distance, a structural rather than data-driven split.

The specialization mechanism comes from LoRA-MCL. There, only the lowest-loss adapter among $K$ is updated, a conditional hard expectation-maximization rule, and with $K=2$ the adapters separated English from French without labels. Early mixture models showed the same positive feedback between routing and expert improvement (Weber et al.).

For diagnostics we take numerical and effective rank from Are we measuring oversmoothing correctly?, which shows that Dirichlet energy misses collapse in trained networks while effective rank tracks accuracy. Backward Oversmoothing acts as a constraint: deep GNNs stall when the final layer converges early. We keep depths moderate so that this effect does not confound frequency specialization.

Research Program

Each layer holds $K$ filter experts. Expert $k$ applies a Chebyshev polynomial filter $h_k(\tilde L)=\sum_{j=0}^{J}\theta_{k,j}T_j(\tilde L)$. Here $\tilde L$ is the rescaled normalized graph Laplacian, $T_j$ is the $j$-th Chebyshev polynomial, and $\theta_{k,j}$ are expert $k$'s learnable coefficients. A polynomial of order $J$ can express low-pass, high-pass, or band-pass responses, so nothing about an expert's frequency role is built in.

All experts share their feature transform and start with identical $\theta$ plus small noise. For each labeled node $v$, the training update is a relaxed winner-takes-all rule:

$$ \mathcal{L}=\sum_{v}\Big[(1-\epsilon)\,\ell_{v,k^*(v)}+\tfrac{\epsilon}{K-1}\sum_{k\neq k^*(v)}\ell_{v,k}\Big]+\lambda\,\mathrm{CE}\big(g(x_v,\Delta_v),\,k^*(v)\big). $$

In this objective, $\ell_{v,k}$ is node $v$'s classification loss under expert $k$, and $k^*(v)=\arg\min_k \ell_{v,k}$ is the winning expert. The small $\epsilon$ passes some gradient to losing experts so they cannot die. The gate $g$ is trained by cross-entropy to predict the winner from the node's features $x_v$ and the mean feature difference to its neighbors $\Delta_v$. At test time the gate alone chooses the expert.

The gate needs information that leaks local homophily, which $\Delta_v$ provides cheaply. Labels are scarce in transductive splits, so we also test a variant in which unlabeled nodes take part in the competition through a feature-reconstruction loss.

The analysis comes in three parts. Before adding experts, we measure frequency gradient interference in a single-filter model. We estimate local homophily on training nodes, split them into low- and high-homophily groups, and compute the cosine between the two groups' gradients on $\theta$ over training. After training, we plot each expert's learned frequency response $h_k(\lambda)$ against the eigenvalue $\lambda$ to see whether low-pass and high-pass experts emerge. We also correlate gate assignments on held-out nodes with their true local homophily. Throughout, we track effective rank of node representations within each expert's assigned group, to check that specialization does not trade one collapse for another.

Baselines

The naive reference point is a plain GCN at the same depth. GSAN reports this GCN at 28.2% on Chameleon (GSAN). We rerun it ourselves on identical splits.

The strongest competitor on heterophilous graphs is GSAN, the existing method closest to node-wise frequency selection. It reports 61.2% accuracy on Chameleon. On Squirrel the strongest number in our file is Ordered GNN at 62.44% (Ordered GNN). FB-GNN reports a gain of roughly 10% over GCN on Texas and Cornell but gives no absolute number in our file. On ogbn-arxiv the reference is the improved vanilla GCN at 72.28% (Old can be Gold). We found no reported result on Lacuna for arXiv-year or Amazon Ratings with any of these methods.

All baselines will be rerun in our codebase with the same splits, the same hidden width, and a parameter count matched within 10%. Each method gets the same random-search budget of 60 configurations and is scored by mean and standard deviation over 10 seeds. Beating a baseline means a gain larger than two pooled standard deviations on at least two heterophilous datasets with no significant loss on Cora, CiteSeer, or PubMed.

The ablation that isolates the key ingredient is the same architecture with soft softmax routing instead of competitive updates. It has identical parameters and identical gate inputs. If the soft version matches the competitive one, emergence through competition is not the cause of any gain. We also run a single Chebyshev filter widened to the same parameter count, to rule out capacity as the explanation.

Why not just…?

Why not just add residual connections and batch normalization? These fix depth collapse, as the 1,024-layer results in The Oversmoothing Fallacy show. They do not change the fact that every node passes through the same filter. Our hypothesis concerns mixed-frequency graphs at ordinary depth, so we include residual-plus-batch-normalization GCN as a baseline at depths 2 through 8. If it matches our method on graphs where we measure strong gradient interference, our claim is wrong.

Why not just use FB-GNN or GSAN? FB-GNN mixes its two channels with dataset-wide scalars and cannot treat a cluster node differently from a boundary node. GSAN is node-wise and is a genuine threat. Its menu of filters is fixed, however, and must be precomputed at several scales for every node. The test is direct. If GSAN's accuracy equals ours and our learned experts merely reproduce GSAN's low-pass and band-pass shapes, then our contribution is a cheaper reparameterization rather than better accuracy, and we will report it that way.

Why not just route each node by its estimated local homophily to a fixed low-pass or high-pass filter? This is the most dangerous alternative, because it hand-codes the split we hope emerges. We build it explicitly. A small classifier predicts local homophily from $(x_v,\Delta_v)$ and hard-routes each node to FB-GNN's low-pass or high-pass operator. We also run an oracle version that uses true homophily computed from all labels. If the predicted-homophily router matches competitive experts, emergence adds nothing on these benchmarks. Competitive experts can win only if useful specializations exist beyond a single homophily axis, such as band-pass responses or more than two groups. The frequency-response plots for $K=3$ and $K=4$ will show whether such experts form.

Decisive Experiments

The first check runs on synthetic graphs. We plant regions of known, differing homophily with Gaussian class features, which gives a ground-truth node partition. Starting from identical initialization with $K=2$, the experts' frequency responses must diverge into opposite-sign slopes, and gate assignments must recover the planted regions well above chance. If they do not diverge, the mechanism fails at its foundation.

The second check is predictive. Across Cora, CiteSeer, PubMed, Chameleon, Squirrel, Actor, Texas, Cornell, Wisconsin, Amazon Ratings, arXiv-year and ogbn-arxiv, the pre-measured gradient conflict between homophily groups should rank datasets by the gain of competitive experts over a single filter. We expect a clearly positive rank correlation. A null correlation would mean that any gains do not come from the interference we describe.

The third check compares competitive experts against soft routing, the homophily router, the oracle router, GSAN, and Ordered GNN under the protocol above. We additionally report effective rank per expert group to confirm that specialization preserves representation diversity.

The outcome is informative in every case. If competitive experts beat the homophily router, graphs contain frequency structure that hand-designed filters miss. If the router matches them, the gradient-interference diagnosis still stands, and it tells practitioners that a cheap homophily estimate is all the routing they need.