Graph neural networks classify a node by repeatedly averaging its features with its neighbors'. After many rounds of averaging, every node looks the same. This failure is called over-smoothing, and it is usually treated as a forward-pass problem: the averaging erases the differences between nodes. There is a second way it can happen that gets less attention. The averaging operator also shapes the gradients that train the feature-extracting layers. If those gradients are themselves smoothed, the model can learn to produce features that are already low-dimensional before any averaging takes place. This note asks a narrow question. When node features and graph structure are processed as two separate streams and merged only at the end (late fusion, the multimodal-learning practice of giving each input type its own encoder), should gradients from the structure stream be allowed to flow back into the feature encoder at all?
The proposal is to build and test a gradient-isolated late-fusion graph model. A feature encoder is trained only on its own node-level loss. A structure branch propagates a detached copy of those features through several low-pass and high-pass graph filters. A per-node fusion head then merges the two streams using calibrated confidences. The central empirical claim is that end-to-end training through a propagation operator lowers the effective dimensionality of the feature encoder's output, and that blocking this gradient path preserves node distinctiveness at large propagation depth and on graphs whose neighbors tend to have different labels. The note is also explicit about when the claim should fail: at very low label rates, gradients flowing through propagation carry useful supervision to unlabeled nodes.
Core Claim and How It Differs From Prior Work
Several papers in the file motivate the claim. Backward Oversmoothing identifies smoothing in the training dynamics of deep GNNs, not only in their outputs. Old can be Gold shows that deep vanilla GCN failures are largely optimization failures: with degree-aware initialization and gradient-guided skip connections, it reaches about 80% on Cora at 64 layers and 72.28% on OGBN-ArXiv. Are we measuring oversmoothing correctly? supplies the measurement tool. It shows that effective rank (a continuous count of how many feature dimensions carry variance) tracks accuracy loss with average Pearson correlation 0.91, while Dirichlet energy (a measure of how much features differ across edges) does not. The same paper warns that high rank does not guarantee good predictions. That warning constrains the design, so every rank measurement below is paired with a linear-probe accuracy.
The closest method is APPNP from Predict then Propagate. APPNP runs an MLP on node features and then propagates its predictions with Personalized PageRank, a random walk that returns to the starting node with probability $\alpha$. APPNP decouples the forward computation but not the backward one. Its MLP is trained through the propagation operator, so the MLP receives neighbor-averaged gradients. Our proposal keeps APPNP's forward idea and cuts that backward path. It also replaces APPNP's single global $\alpha$ with a per-node fusion weight.
GCNII from Simple and Deep Graph Convolutional Networks mixes the initial representation into every layer. It is effectively intermediate fusion with full gradient sharing.
Two methods add high-pass information inside every layer. FB-GNN from Complete the Missing Half adds a high-pass channel with per-layer scalar weights. ALaGCN from When Do GNNs Work gates aggregation per node using neighborhood entropy and center-neighbor similarity. We reuse both signals, but only at the late fusion stage.
UDPD from Bridging Structure and Semantics is the nearest dual-path design. It runs separate semantic and structural paths with an uncertainty gate. However, it relies on LLM keyword extraction and iterative diffusion sampling, it lets the paths interact during diffusion, it does not measure rank collapse, and it leaves heterophily to future work.
The lesson we take from late fusion in other domains comes from A Multi-Stream Fusion Network for Image Splicing Localization. That work found that a dominant stream suppresses subtle ones unless the streams are merged only at the bottleneck, and that connections must be asymmetric across streams. We translate "asymmetric connections" into "asymmetric gradients."
A caveat on novelty: combining separate ego-node and neighbor embeddings is a familiar move in heterophily work beyond this file. The new content here is not the two-branch forward architecture. It is the backward-path hypothesis and the rank-based evidence for or against it.
Research Program
The model has three parts.
The feature encoder $f$ is an MLP on node features $X$. It is trained with cross-entropy on labeled nodes only, so it never sees the adjacency matrix.
The structure branch computes a filter bank from the detached feature output $\bar H = \mathrm{sg}(f(X))$, where $\mathrm{sg}$ blocks gradients. The bank contains Personalized PageRank propagations at several depths $K$, together with the high-pass residual $\bar H - \tilde A \bar H$, where $\tilde A$ is the normalized adjacency matrix. A small learned readout on this bank produces structure-branch logits.
The fusion head outputs, for each node $u$,
$$p_u = \lambda_u\, \mathrm{softmax}(z^{f}_u / T_f) + (1-\lambda_u)\, \mathrm{softmax}(z^{g}_u / T_g),$$
where $z^f_u$ and $z^g_u$ are the two branches' logits and $T_f, T_g$ are temperatures fitted on validation nodes. The temperatures address the branch-calibration weakness of late fusion noted in the file. The weight $\lambda_u \in [0,1]$ comes from a tiny network whose inputs are the feature branch's predictive entropy, the structure branch's entropy, and the center-neighbor similarity of $\bar H$. That last signal is borrowed from ALaGCN.
The analysis half matters as much as the model. For every architecture, we record three quantities at each propagation depth: the effective rank and numerical rank of the feature encoder's output, the effective rank of the final representation, and the accuracy of a linear probe trained on the frozen feature-encoder output. The key comparison is the same architecture trained two ways, with and without $\mathrm{sg}$. If backward over-smoothing is real in this setting, the feature-encoder rank under joint training should fall as $K$ grows, while under isolation it should stay flat by construction.
Baselines
The naive reference points are a feature-only MLP and a plain GCN. The GCN's known collapse to about 26% on Cora by layer 10, reported in Old can be Gold, anchors the depth curves. A second naive point is a post-hoc ensemble: train an MLP and an APPNP model separately and average their logits. This is the simplest possible late fusion, and it tests whether our fusion head adds anything.
APPNP is the most direct competitor on the backward-path question, because it shares our forward structure. We run it at identical propagation depths, with $\alpha$ tuned under the same budget. The best depth-robust competitor on homophilic graphs is the improved vanilla GCN from Old can be Gold, with a reported 72.28% accuracy on OGBN-ArXiv and about 80% on Cora at 64 layers. On heterophilic graphs, the strongest competitors are GCNII*, reported to beat APPNP by over 12% on Wisconsin, and CMGNN from Understanding and Enhancing Message Passing on Heterophilic Graphs via Compatibility Matrix, which reports an average rank of 1.3 on heterophilic datasets. The file gives no absolute accuracies for these on Texas, Cornell, or Wisconsin, so we will rerun both. We treat UDPD's 78.55% on OGBN-ArXiv as a reference only, because it uses LLM-processed text and is not a like-for-like comparison.
Fairness rules apply to every method. All methods use the same splits, including standard splits and 1%, 5%, and full label rates. All use the same raw features, with no LLM features for anyone, the same hyperparameter search budget, and ten seeds.
Three ablations isolate the key ingredient: joint training with gradients flowing into $f$, a scalar global $\lambda$ in place of the per-node gate, and a filter bank without the high-pass term.
Beating the baselines has a specific meaning. We need statistically significant gains over APPNP and the joint-training ablation at depths of 16 and above, and on heterophilic benchmarks. We also need to match the improved vanilla GCN on OGBN-ArXiv within noise, at lower training cost.
Why not just…?
Why not just use APPNP? On the forward side, APPNP already is the decoupled design, so the entire bet is on the backward path. The joint-versus-isolated ablation settles this directly. If APPNP's MLP keeps its effective rank and probe accuracy at large $K$, the hypothesis is wrong, and the honest conclusion is that APPNP's teleport term is sufficient. The only remaining advantage would then be the per-node gate on heterophilic graphs, where the 12% Wisconsin gap reported by GCNII suggests APPNP's single global $\alpha$ is too rigid.
Why not just average an MLP and a GNN? This is the most serious threat, because it is gradient-isolated by construction. It lacks the per-node gate and the high-pass filter, and it does not calibrate the two branches against each other. We include it as a baseline. If it matches our model, the correct conclusion is that isolation matters and the fusion machinery does not, and we would report exactly that.
Why not just use GCNII or PairNorm? Both prevent collapse at depth. However, PairNorm reports that it does not consistently improve peak accuracy, and GCNII still passes smoothed gradients through every transformation. Our rank diagnostics apply to both methods, so we can check whether they preserve accuracy despite losing rank in the feature layers, or because they avoid that loss.
Why not keep gradients flowing but add an auxiliary MLP loss? This is the standard multimodal remedy for one input type dominating training, and it may recover most of the benefit. We include it as a fourth training mode, between fully joint and fully isolated training.
Decisive Checks
We will run the experiments on Cora, Citeseer, PubMed, OGBN-ArXiv, Texas, Cornell, Wisconsin, Chameleon, Squirrel, and Amazon-Ratings. Two outcomes decide the proposal.
The first is whether, under joint training, the feature encoder's effective rank and probe accuracy decline with propagation depth while they stay stable under isolation, and whether that gap predicts the accuracy gap across datasets. A null result here falsifies the core claim.
The second is the label-rate sweep. We expect isolation to lose at the 1% label rate on homophilic graphs, where ALaGCN's reported 73.3% on Cora at that rate sets the bar. The reason is that back-propagated neighbor gradients carry supervision to unlabeled nodes. We expect isolation to win at higher label rates and under heterophily. If isolation loses everywhere, the idea should be dropped. If it wins where predicted, the result tells practitioners when a graph's structure should be allowed to train its feature encoder.
References
- Backward Oversmoothing: why is it hard to train deep Graph Neural Networks?
- Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again
- Are we measuring oversmoothing in graph neural networks correctly?
- Predict then Propagate: Graph Neural Networks meet Personalized PageRank
- Simple and Deep Graph Convolutional Networks
- Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks
- When Do GNNs Work: Understanding and Improving Neighborhood Aggregation
- Bridging Structure and Semantics: Uncertainty-Modulated Dual-Path Diffusion for Robust Text-Attributed Graph Learning
- A Multi-Stream Fusion Network for Image Splicing Localization
- Understanding and Enhancing Message Passing on Heterophilic Graphs via Compatibility Matrix
- PairNorm: Tackling Oversmoothing in GNNs