Unlabeled Depth Supervision: Using a Mean Teacher to Keep Deep GNNs Trainable

Graph neural networks (GNNs) compute each node's representation by repeatedly averaging it with its neighbors. Stack many of these averaging layers and every node starts to look the same, a failure called over-smoothing. Recent theory shows the failure is also an optimization trap. The final classifier quickly fits the few labeled nodes using already-blurred features. Its gradient then vanishes, so the earlier layers stop learning and stay near their random initialization (Backward Oversmoothing). This matters most when labels are scarce, because a handful of labeled nodes is easy for the classifier to fit and the trap closes early.

We propose to hold the trap open with a Mean Teacher: a slowly updated copy of the network, whose weights are an exponential moving average (EMA) of the trained network's weights. This teacher produces soft predictions for every unlabeled node, and those predictions become targets for small auxiliary classifiers attached at several depths of the trained ("student") network. The student sees a heavily message-dropped graph; the teacher sees the clean graph. The student therefore keeps receiving informative, node-specific error signals at every depth long after the labeled loss has been driven to zero. In two sentences: deep supervision normally needs labels at every layer, and the EMA teacher supplies those labels for free on unlabeled nodes. We test whether this prevents the gradient freeze that makes deep GNNs untrainable, and whether that matters on tasks where depth actually helps.

Core Claim

Our hypothesis is that, in the scarce-label regime, much of the accuracy loss of deep message-passing networks comes from backward oversmoothing, and that a consistency loss against an EMA teacher's depth-ensembled predictions removes it. It can do so only if the targets differ from node to node at depths where the student's own features have already blurred. We therefore make the teacher's target the confidence-weighted average of its auxiliary heads across all depths, so shallow, still-discriminative heads feed targets to deep student heads. Formally, with student heads $f^{(l)}_\theta$ at depths $l \in \mathcal{L}$, teacher target $q_v = \mathrm{sharpen}\big(\tfrac{1}{|\mathcal{L}|}\sum_{l} f^{(l)}_{\theta'}(v; G)\big)$ for unlabeled node $v$ on the clean graph $G$, and a confidence mask $m_v = \mathbb{1}[\max q_v > \tau]$, the objective is

$$\mathcal{L} = \sum_{l\in\mathcal{L}} \Big[\mathrm{CE}\big(f^{(l)}_\theta(\tilde G), y\big)_{\text{labeled}} + \lambda \sum_{v\ \text{unlabeled}} m_v\, \mathrm{KL}\big(q_v \,\|\, f^{(l)}_\theta(v;\tilde G)\big)\Big],$$

where $\tilde G$ is the graph with messages randomly dropped, $\lambda$ weights the consistency term, $\theta$ are the student weights, and the teacher weights follow $\theta'_t=\alpha\theta'_{t-1}+(1-\alpha)\theta_t$.

The new element is not the Mean Teacher and not deep supervision taken separately. It is the use of teacher-generated, cross-depth targets on unlabeled nodes as a tool against gradient collapse, together with instrumentation that checks whether the gradient collapse is actually what gets fixed.

Relation to Prior Work

Two papers motivate the gap. Backward Oversmoothing proves that backpropagated gradients converge exponentially to an uninformative constant with depth, and that the fast-fitting classifier strands earlier layers at spurious stationary points. It does not test any remedy based on a teacher or a consistency loss. Are we measuring oversmoothing in graph neural networks correctly? shows that effective rank of the feature matrix tracks accuracy (Pearson 0.91) while Dirichlet energy does not. We adopt effective rank as our representation diagnostic, not as a training objective.

Two papers supply the method, and each lacks what the other has. Improving the Training of Graph Neural Networks with Consistency Regularization introduces SCR-m, a dropout-based EMA-teacher consistency loss with confidence masking and sharpening. It uses shallow or decoupled models (GraphSAGE, GAMLP) and never measures behavior across depth. Addressing Over-Smoothing in Graph Neural Networks via Deep Supervision (DSGNN) puts classifiers at every layer and averages their losses. It needs ground-truth labels at each head, so under scarce labels its intermediate heads see almost no signal. Our method is DSGNN's architecture trained with SCR-m's unlabeled signal, with targets that cross depths.

Other work sits nearby but differs. Tackling Oversmoothing of GNNs with Contrastive Learning (Deeper-GXX) reaches 60 layers with decaying residuals and a neighbor/non-neighbor contrastive loss. It is single-network and relies on negative pairs, which it concedes scale poorly; our loss needs no negatives. FreeKD distills between two shallow GNNs under a reinforcement-learning arbiter, explicitly avoiding deep models. We keep one deep model and a passive EMA copy.

Two papers constrain the design. Shedding Light on Random Dropping and Oversmoothing shows that DropEdge and DropMessage act only as training-time augmentation and do not reduce test-time over-smoothing. We therefore use dropping only as the student's perturbation and never claim it fixes smoothing by itself. Position: Don't be Afraid of Over-Smoothing And Over-Squashing argues that deep models usually lose to 2–4-layer ones because distant neighbors are uninformative, not because features collapse. This directly threatens the premise. We answer it by restricting claims to regimes where depth has been shown to help and by running an experiment that separates the two explanations.

There is also a skeptical reading of EMA itself, raised around Consistency Training with Exponential Moving Average Teachers. Parameter averaging is a low-pass filter over training time and might simply entrench collapsed solutions. Note that EMA smooths across training steps, not across graph nodes. Whether it helps or hurts deep GNNs is an empirical question that our gradient measurements answer directly.

Research Program

We will build the method on two backbones: a plain deep GCN, the setting where backward oversmoothing is proven, and GCNII (Simple and Deep Graph Convolutional Networks). Depths are 8, 16, 32, and 64, with auxiliary heads every four layers. We also build a measurement harness that logs, per layer and per epoch, three quantities. The first is the gradient norm at each layer relative to the last, which directly tests the backward oversmoothing mechanism. The second is effective rank and numerical rank of the features. The third is the across-node variance of the backpropagated gradient, which is the quantity the theory says goes to a constant.

The analysis question is whether accuracy gains coincide with non-vanishing early-layer gradients, with higher feature rank, or with neither. Neither would mean the gain comes from ordinary semi-supervised regularization. Because the approach works only if teacher targets stay node-discriminative, we also report the across-node entropy of teacher targets at each depth over training. If deep heads' targets collapse along with the student, the mechanism has failed and we will say so.

Baselines

The naive references are two plain GCNs trained with labeled cross-entropy only: a 2-layer GCN (the tuned shallow model the position paper says usually wins) and a deep GCN at matched depth. The deep GCN is expected to collapse, as the reported 29.62% for a 60-layer GCN on Cora shows (Tackling Oversmoothing of GNNs with Contrastive Learning). Beating the deep GCN is trivial. The real bar is the shallow GCN on the tasks where we claim depth matters.

The strongest existing deep method in the file is Deeper-GXX, reported at 80.59% test accuracy on Cora at 60 layers (Tackling Oversmoothing of GNNs with Contrastive Learning). Learn2Drop is reported at 69.3% with a 64-layer GCN on Cora (Shedding Light on Random Dropping and Oversmoothing). GCNII is the strongest architectural fix; no Cora accuracy is given in the file, so we rerun it. The file does not guarantee these papers share splits with each other or with our scarce-label splits. We therefore rerun Deeper-GXX, GCNII, and Learn2Drop from their descriptions on identical splits, seeds (ten per cell), epochs, and a matched hyperparameter search budget, the same number of trials for every method. The reported numbers serve only as sanity checks on our reimplementation. On large graphs the strongest semi-supervised reference is SCR-m, reported at 84.62% on ogbn-products with GAMLP (Improving the Training of Graph Neural Networks with Consistency Regularization). For ogbn-arxiv, no reported result was found on Lacuna for any of these methods.

The most dangerous baseline is the obvious combination of existing parts, and we must beat it: DSGNN heads trained on labeled nodes, plus SCR-m applied only at the final output. The ablations that isolate the key ingredient are the following. Depth-matched targets, where teacher head $l$ supervises student head $l$, test whether crossing depths matters. A detached-student target without EMA (the student's own clean-graph prediction) tests whether the slow teacher matters. Removing the confidence mask tests confirmation bias.

"Beating" means that at depth 32 or more, in the depth-favorable regimes below, our method exceeds both the combined DSGNN+SCR-m baseline and GCNII by more than the across-seed standard deviation. It must also be at least on par with the tuned 2-layer GCN. On standard fully featured Cora, we only need to not lose relative to the shallow model.

Why not just…?

Why not just use GCNII? GCNII fixes the forward pass with residuals to the input features, and in the position paper's re-evaluation it rarely beat a tuned shallow GCN (Position: Don't be Afraid of Over-Smoothing And Over-Squashing). The backward-oversmoothing analysis says skip connections do not fully remove the gradient stall. Still, GCNII is a real threat. We apply our loss on top of GCNII as well as on a plain GCN. If GCNII plus our loss does not beat GCNII alone, the method adds nothing beyond architecture.

Why not just slow down or freeze the classifier head? If backward oversmoothing comes from the head converging too fast, the cheapest remedy is a smaller learning rate on the head, or delayed head training. This is the alternative most likely to match us, so we run it as a first-class baseline, tuning the head learning-rate ratio over the same budget. If it recovers early-layer gradient norms and accuracy as well as our method does, the conclusion is that the teacher is unnecessary. We would then report the proposal as a negative result about the need for a teacher, not as a method.

Why not just stay shallow? On most fully featured benchmarks, shallow wins (Position: Don't be Afraid of Over-Smoothing And Over-Squashing). We therefore make no claim there. Our claims are limited to settings where deep models were shown to help: missing-feature node classification, where DSGNN and Deeper-GXX report deep models repairing features from distant context, and molecular regression, where DSGNN reports depth gains.

Why not just combine DSGNN with SCR-m, or use DropMessage alone? DropMessage alone is augmentation and does not affect test-time collapse (Shedding Light on Random Dropping and Oversmoothing). The DSGNN+SCR-m combination has no unlabeled signal at intermediate heads, which are exactly the heads that need it under scarce labels. The prediction that separates the two is gradient-level: at deep layers our method should keep early-layer gradient norms orders of magnitude above the combination's. If it does not, the added mechanism is not doing what we claim.

Proposed Experiments

For node classification we use Cora, CiteSeer, and PubMed under three conditions: the standard sparse-label split, a reduced-label split, and the missing-feature setting used by DSGNN, where test-node features are removed. We add Chameleon and Squirrel for heterophily and ogbn-arxiv for scale. For graph-level tasks we use ESOL and Lipophilicity with DSGNN's protocol. If the method survives ogbn-arxiv, we will run a single scaling check on ogbn-products against SCR-m's 84.62% (Improving the Training of Graph Neural Networks with Consistency Regularization). Metrics are test accuracy (RMSE for regression), per-layer effective rank, and per-layer gradient-norm ratios, all averaged over ten seeds.

The first decisive check is mechanistic. On a 64-layer plain GCN on Cora with scarce labels, plot the gradient-norm ratio between layer 1 and the last layer over training for the naive deep GCN, the head-learning-rate baseline, DSGNN+SCR-m, and our method. The hypothesis predicts that only methods which keep this ratio from collapsing recover accuracy.

The second decisive check separates collapse from noisy distant neighbors. On missing-feature PubMed, we compare against a receptive-field-matched control: a shallow model given multi-hop features at the same radius. If our deep model's gain disappears against that control, the benefit comes from reaching farther, not from anti-smoothing. That outcome supports the position paper's account and would reduce our claim to "the teacher makes large receptive fields trainable."

The third check tests the EMA concern. We sweep $\alpha$ from 0.9 to 0.999. If the highest $\alpha$ lowers feature rank relative to the detached-student ablation, the parameter-smoothing worry is confirmed and we will report that the teacher must stay fast in deep GNNs.