Graph neural networks (GNNs) classify nodes by repeatedly averaging each node's features with its neighbors'. That makes them slow to serve, because every prediction requires fetching a multi-hop neighborhood from memory. The standard fix is knowledge distillation: train a plain multilayer perceptron (MLP), which looks only at a node's own features, to imitate the GNN's outputs. Modern versions of this fix have grown into multi-stage pipelines. The most accurate one, Adaptive Hierarchical Knowledge Distillation from GNNs to MLPs, trains the teacher GNN, then runs a neural architecture search with an LSTM controller to pick the student's depth and width, and then aligns the student to the teacher layer by layer. It is motivated by an observed "drift" between teacher and student representations as depth grows, which it attributes to a generic capacity mismatch.
We propose a different reading of that drift. It is mostly over-smoothing: repeated averaging makes node representations collapse toward a few directions, so a deep GNN's later layers carry little usable information. Are we measuring oversmoothing in graph neural networks correctly? shows that this collapse is captured by the effective rank of the node-representation matrix, and that effective rank tracks accuracy with an average Pearson correlation of 0.91. If the reading is right, two things follow. First, the architecture search can be replaced by one cheap measurement of the teacher's per-layer effective rank, which tells us how wide the student must be and which teacher layers are worth imitating. Second, forcing the student to match teacher layers beyond the point of collapse should actively hurt it. The result would be a pipeline with one teacher run and one student run, plus an explanation of when GNN-to-MLP distillation fails.
Core Claim
The claim is that the teacher's effective rank, measured per layer, is a sufficient statistic for the two decisions that current pipelines make by search or by hand: how wide the student MLP must be, and which intermediate teacher layers it should be aligned to. For a teacher layer with node representations $H_l \in \mathbb{R}^{n \times d}$ ($n$ nodes, $d$ hidden units), let $\sigma_1,\dots,\sigma_d$ be its singular values and $p_i = \sigma_i / \sum_j \sigma_j$. The effective rank is
$$ \mathrm{Erank}(H_l) = \exp\Big(-\sum_i p_i \log p_i\Big). $$
We pick the alignment cutoff $l^\*$ as the last layer before Erank falls below a fixed fraction of its peak. We set the student hidden width to $d_s = \lceil c \cdot \max_{l \le l^\*} \mathrm{Erank}(H_l) \rceil$, where $c$ is a single slack constant fixed once across all datasets.
Computing this needs one forward pass and a thin singular value decomposition, which costs $O(nd^2)$ and can be run on a node subsample. The student is then trained in a single run on three things together, in the one-step style that Simplified TinyBERT found halves training time in text retrieval: soft labels, hard labels, and MSE alignment only to layers $l \le l^\*$.
The falsifiable prediction concerns deep teachers. When a teacher over-smooths, all-layer alignment as done in AHKD should do worse than truncated alignment. The student's own effective rank should fall in step with the teacher layers it is forced to copy.
Research Program
The first part is diagnostic. It asks whether teacher over-smoothing actually propagates into students, which nobody in the file has measured. We train GCN, GraphSAGE and GAT teachers at depths 2, 4, 8 and 16. We distill each into students with three methods: GLNN-style soft-label matching (Graph-less Neural Networks), AHKD-style all-layer alignment, and our truncated alignment. For each run we record per-layer Erank for both teacher and student, alongside accuracy. The questions are whether the student's final-layer Erank tracks the teacher's, and whether the accuracy gap between student and teacher is predicted by the teacher's rank collapse better than by teacher depth alone. We also record Dirichlet energy, a common smoothness measure, to test on the distillation setting the finding that it is a poor over-smoothing signal in trained networks.
The second part tests the replacement rule. For every teacher, we compare the Erank-chosen width and cutoff against an exhaustive grid over student widths and alignment cutoffs. We report where the rule lands relative to the best grid point, and how much end-to-end wall-clock it saves.
The third part stresses the claim where it is most likely to break: graphs where linked nodes tend to have different labels, called heterophilic graphs. Here averaging is the wrong operation, and Complete the Missing Half shows that a separate high-pass "diversification" channel improves accuracy on Texas and Cornell by roughly 10%. On these graphs we also use FB-GNN teachers and measure Erank separately for the low-pass and high-pass channels. This tells us whether the rule needs to be per-channel. That is a direct test of whether a single scalar rank misses information carried in high frequencies.
The main known risk is that high rank does not guarantee useful information, since random noise has maximal rank, as the over-smoothing measurement paper itself notes. We address it in two ways. Rank is only used to set an upper budget and a cutoff, never as a target to maximize. And an ablation adds an explicit rank-floor penalty on the student, to test whether pushing rank up helps or simply injects noise.
Baselines
The naive reference points are a plain MLP trained on labels and the teacher GNN itself. These bound the gain and the inference cost. On ogbn-arxiv, AHKD reports a plain MLP at 53.88% accuracy.
Vanilla soft-label distillation, as in GLNN, is the simplest real competitor and is the default in practice. AHKD reports GLNN at 64.88% on ogbn-arxiv.
The strongest existing method is Adaptive Hierarchical Knowledge Distillation from GNNs to MLPs. It reports 71.40% on ogbn-arxiv and an 11x inference speedup over a 3-layer GraphSAGE on Flickr (82ms versus 953ms). It is the strongest competitor because it targets exactly the depth-dependent drift we reinterpret. We will rerun it with the same teacher checkpoints as our method, and we will count its search time in its pipeline cost.
KRD adds an offline profiling stage: it perturbs node features with noise to score how reliable each node's teacher output is. It reports a 12.62% average gain over plain MLPs and a 2.16% average gain over teachers. The file gives no per-dataset numbers for it, so we will reproduce it. TGS removes the teacher entirely and instead enforces consistency between neighbors' predictions. It is the stage-free comparison point, and it is especially relevant on heterophilic graphs, where that consistency may itself over-smooth.
All methods share the same splits, the same teacher checkpoints where a teacher is used, and the same number of tuning trials per method on validation accuracy. Wall-clock is measured end to end on identical hardware. Beating AHKD means matching or exceeding its accuracy within seed noise, while using no search and lower total pipeline time. On deep teachers we additionally expect a strictly higher accuracy than AHKD's all-layer alignment.
Three ablations isolate the key ingredients. Erank-chosen width versus a fixed width tests the width rule. Truncated versus all-layer alignment at a fixed width tests the cutoff rule. Selecting the cutoff by Dirichlet energy, or at random, instead of by Erank tests whether the measure matters.
Why not just…?
Why not just grid-search the student width and alignment cutoff? This is the most serious alternative, because training an MLP is cheap and a small grid may recover whatever the Erank rule finds. We do not argue around it. The grid is run in full as a reference in the second part of the program. The rule earns its place only if it lands within seed noise of the best grid point at a fraction of the cost, and if the cutoff it picks is explained by measured rank collapse. If the grid consistently wins by a clear margin, the paper's contribution narrows to the diagnostic finding, and we will report it that way.
Why not just use a teacher that doesn't over-smooth? Tackling Oversmoothing of GNNs with Contrastive Learning keeps a 60-layer model at 80.59% on Cora, where a 60-layer GCN falls to 29.62%. Distilling such a teacher with vanilla soft labels could make our rule unnecessary. The cost is a heavier teacher stage, with deep residual stacks and a pairwise contrastive loss. We include Deeper-GXX and FB-GNN teachers with GLNN-style distillation as explicit arms. If they match our method at comparable total cost, we report that our approach is mainly useful for standard teachers.
Why not just add residual connections to the teacher? The over-smoothing measurement paper reports that rank still decays and accuracy drops around 10 layers even with normalization and residual connections. This suggests residuals delay collapse but do not remove the need to know where it happens. Our depth sweep includes residual GCN teachers, so the experiments show directly whether the cutoff simply moves deeper or disappears.
Why not just prune or quantize the GNN? GLNN reports that graph-free MLP students are 14x to 27x faster than pruned or quantized GNNs. The reason is that the bottleneck is fetching neighbors, not arithmetic, so we treat this as settled by prior evidence and do not rerun it.
Proposed Experiments
The homophilic benchmarks, where linked nodes tend to share labels, are Cora, Citeseer, Pubmed, Amazon-Computer, Amazon-Photo, Flickr, ogbn-arxiv and ogbn-products. The heterophilic benchmarks are Texas, Cornell, Squirrel and Chameleon. All are evaluated in both transductive and inductive settings, the latter being where GLNN reports its weakest results. Metrics are test accuracy over ten seeds, end-to-end pipeline wall-clock, and inference latency. We also report per-layer Erank and the correlation between the student's rank gap and its accuracy gap.
Three decisive checks determine the outcome. The core hypothesis is falsified if, on teachers of depth 8 or more, all-layer alignment matches or beats truncated alignment. It is also falsified if student Erank does not follow the aligned teacher layers. The width rule fails if it lands outside seed noise of the best grid point on most datasets. The practical claim stands only if we reach AHKD's reported 71.40% on ogbn-arxiv, in our own reproduction with matched teachers, with one student run and no search. On heterophilic graphs, a per-channel rank rule outperforming the single-scalar rule would show that frequency-specific rank, not total rank, is the right signal. That would be a finding in its own right.
References
- Adaptive Hierarchical Knowledge Distillation from GNNs to MLPs
- Are we measuring oversmoothing in graph neural networks correctly?
- Simplified TinyBERT: Knowledge Distillation for Document Retrieval
- Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation
- Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks
- Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs
- A Teacher-Free Graph Knowledge Distillation Framework With Dual Self-Distillation
- Tackling Oversmoothing of GNNs with Contrastive Learning