A common way to get fast predictions on graph data is to train a graph model as a "teacher," let it label every node, and then train a plain multi-layer perceptron (MLP) "student" to copy those labels from node features alone. The student never looks at the graph, so it runs orders of magnitude faster and still works on new nodes that arrive with no edges. The weak point is that the student copies the teacher's mistakes as faithfully as its correct answers. This matters most when only a handful of nodes carry human labels, which is also when graph methods are most useful. In that regime the teacher is often a label propagation process, meaning labels spread from a few seed nodes to their neighbors, and its errors are not random. They come in whole regions: clusters far from any seed, or pockets where the graph's edges connect nodes of different classes. Existing distillation either trusts every teacher label equally or estimates trust one node at a time from the teacher's confidence score.
This proposal uses the graph twice. The first use is standard: the graph produces the teacher's labels. The second use is new: the graph also estimates where those labels are wrong. We hold out some seed nodes, record whether the teacher recovers their labels, and spread these correct/incorrect indicators over the same graph by label propagation. The result is a per-node reliability estimate. It enters a noise-aware distillation loss that is only needed at training time, so the deployed student stays completely graph-free. The testable claim is that graph position predicts teacher error better than teacher confidence does. If so, a student trained with graph-conditioned trust should beat both blind copying and confidence-based noise correction at low label rates. If not, the method reduces to an existing one, and we will know.
Core Claim
Teacher errors in graph semi-supervised learning are spatially correlated, so a calibrator that sees graph-derived evidence should flag them much more accurately than one that sees only the teacher's output margin. Converting that better error estimate into forward loss adjustment should give a graph-free student that is more accurate and better calibrated, especially on nodes far from the labeled seeds and on isolated nodes. The graph acts as privileged information: it is used only inside the training loss and never at inference.
Relation to Prior Work
Three works establish the pipeline we modify. GLNN distills a message-passing graph network into an MLP by matching its soft labels with KL divergence. It reports an average 12.36% gain over label-only MLPs and 146x to 273x faster inference, but it treats every teacher label as equally trustworthy. GKD uses a modified label propagation teacher with a "remembrance term" to distill into a feature-only student for patients who arrive without graph edges. It changes how labels are propagated, not how much the student should believe them. Teaching Semi-Supervised Classifier via Generalized Distillation distills harmonic-function propagation over privileged features into a kernel learner. It motivates the "graph as privileged information" framing, but it has no noise model and relies on shallow learners. We keep the architecture of all three and change only the loss.
The noise-handling method comes from SLaM. SLaM models the teacher as correct with probability $\alpha(x)$, fits $\alpha$ as a monotone function of the teacher's top-two logit margin using isotonic regression on a validation set, and trains the student with forward loss adjustment. On ImageNet with 5% labels it raises distilled accuracy from 41.67% to 48.1%. SLaM assumes errors are independent across examples and depends on the margin alone. We keep its forward adjustment and replace its reliability estimator with one that uses graph structure. Label Propagation with Weak Supervision supports the premise that propagation quality depends on graph structure: standard propagation covered only about 9% of the SMS dataset because most nodes were disconnected from the seeds. Such regions are exactly where margin-based trust would be misleading, since a propagation score there can be confidently wrong or simply empty.
Two works shape the teacher and the setting. On the Equivalence of Decoupled Graph Convolution Network and Label Propagation shows that decoupled graph networks train a classifier on labels spread by Personalized PageRank, a random-walk-with-restart form of label propagation. This justifies using a propagation-style teacher whose per-node "reach" from the seeds can be measured directly. Scale-Free Graph-Language Models shows that at a 0.03% labeling rate on ogbn-arxiv, propagation over a cosine k-nearest-neighbor (KNN) graph far outperforms naive baselines. That is the extreme low-label regime where teacher errors should be largest and most clustered. Two works constrain the design. Cold Brew and SA-MLP add structure back into the student, through discovered virtual neighbors and adjacency-vector inputs respectively. SA-MLP's structure encoder receives an empty input for true cold-start nodes. We deliberately keep the student a plain feature MLP, so any gain comes from better targets and not from a stronger student.
Research Program
The teacher is a decoupled propagation model: an MLP on features followed by Personalized PageRank propagation, as in PTA/APPNP. We also run a GraphSAGE teacher to show the idea is not tied to one teacher. For the teacher's own labels we use a cosine KNN graph when the native graph is sparse or absent.
Measuring teacher correctness raises a technical problem. Labeled nodes serve as propagation seeds, so the teacher is trivially right on them. We therefore use cross-fitting. The labeled set is split into $F$ folds. For each fold, the teacher is rerun with that fold removed from the seeds, and a correctness indicator $e_j \in \{0,1\}$ is recorded for each held-out node $j$. Every labeled node thus contributes one honest correctness observation, and the final teacher still uses all seeds.
These indicators are sparse, one per labeled node. We spread them to all nodes by solving the same harmonic problem label propagation solves. The unknown reliability surface $r$ minimizes $$\sum_{i,j} W_{ij}(r_i - r_j)^2 \quad \text{subject to } r_j = e_j \text{ on held-out seeds},$$ where $W$ is the graph's edge-weight matrix. This follows the energy view of propagation used in LaplaceNet. The propagated $r_i$ is combined with a small set of local signals in a low-capacity monotone calibrator, fit on the held-out indicators to produce $\alpha_i$. The local signals are the teacher margin, the total propagation mass reaching node $i$ from all seeds, agreement between node $i$'s teacher label and its neighbors' teacher labels, and disagreement between the teacher and a feature-only MLP.
The student $p_S(\cdot \mid x_i)$ sees only node features $x_i$. It is trained, following SLaM, to explain the teacher's label $\hat y_i$ through a noisy channel: $$\mathcal{L} = -\sum_i \log\Big[\alpha_i\, p_S(\hat y_i \mid x_i) + (1-\alpha_i) \sum_{c} p_S(c \mid x_i)\, N(\hat y_i \mid c)\Big].$$ Here $N(\hat y \mid c)$ is the probability that the teacher outputs $\hat y$ when the true class is $c$ and the teacher is wrong. It is estimated from the held-out seeds, restricted to the teacher's top-$k$ classes as in SLaM. The student learns to predict the true class, and the reliability weights determine how much each teacher label counts.
The analysis has two parts. First, we measure how well each reliability estimator predicts teacher error, using AUROC on held-out seeds. The estimators compared are margin only (SLaM), local graph features, and propagated reliability. Second, we measure how much of any student gain is explained by that error-prediction gap. We also characterize failure regimes. On heterophilic graphs, where neighbors tend to have different labels, the smoothness assumption behind propagating reliability may itself fail.
Baselines
The naive reference is an MLP of the student architecture trained only on the labeled nodes with cross-entropy. GLNN reports that this is on average 12.36% worse than a distilled MLP. Any distilled method should clear it by a wide margin; if one does not, it is not transferring graph knowledge.
The strongest existing competitor for graph-free inference is GLNN: the same teacher and the same student, trained by KL matching of soft labels. Soft labels already encode some uncertainty, so this is the fair test of whether explicit noise modeling adds anything. The file gives no numbers for GLNN at very low label rates on these benchmarks, so we will run it ourselves. GKD's remembrance-term propagation teacher distilled with cross-entropy is a second strong competitor. It is the only prior method built specifically for propagation teachers and edgeless test nodes. On its clinical benchmarks it reports 0.667 accuracy on ABIDE and 0.848 on TADPOLE, and we include ABIDE and TADPOLE so we can compare against those published values directly.
The closest method, and the one this proposal must beat, is SLaM applied unchanged to graph data: isotonic calibration on the margin, fit on the same cross-fitted held-out seeds, with the same forward adjustment. It reports 48.1% versus 41.67% for vanilla distillation on ImageNet with 5% labels. No reported result was found on Lacuna for SLaM on graph node classification.
Two ablations isolate the key ingredient. In the first, the calibrator keeps the local graph features but receives no propagated reliability. In the second, reliability is propagated over a degree-preserving shuffled graph. If shuffling does not hurt, the gain is not coming from graph structure. We also include a "filter" baseline that drops teacher labels whose margin falls below a tuned threshold.
For reference, we report the transductive teacher itself, which is not graph-free, and on ogbn-arxiv at 0.03% labels the published graph-dependent SFGL results of 32.51% and 45.70% for SFGL+GPT. For the cold-start split we compare against Cold Brew, which reports up to +2.4% on isolated nodes in open benchmarks.
All methods share the teacher, the student architecture, the label budget, and a fixed hyperparameter search budget per method. Baselines that do not need the held-out folds for calibration may instead use them for early stopping or as extra training labels, so no method gets extra supervision. Beating a baseline means a mean accuracy gain larger than the across-seed standard deviation over at least ten label draws, consistent across datasets, with the largest gains on low-reach and isolated nodes.
Why not just…?
Why not just use the teacher's soft labels, as GLNN does, since they already express uncertainty? Soft labels express the teacher's own confidence, and our premise is that this confidence is miscalibrated in exactly the regions that matter. A seed-starved cluster can receive a confident but wrong label from a single distant seed. Whether that happens is an empirical question. The GLNN comparison and a reliability diagram of teacher confidence stratified by propagation reach will show whether soft labels are already calibrated there. If they are, our gains over GLNN will be small and we will report that.
Why not just filter or down-weight low-confidence teacher labels? This is the cheapest competitor and a real threat, because correlated errors often coincide with low margins. Filtering has two drawbacks. It discards the regions that most need supervision, and it cannot catch confident errors. The tuned-threshold baseline tests this directly. If it matches our method, graph-conditioned trust adds nothing beyond a heuristic.
Why not just apply SLaM as is? This is the most serious alternative, because our loss is SLaM's loss. The whole contribution is the reliability estimate. The decisive measurement is error-prediction AUROC. If propagated and graph-derived reliability do not beat margin-only isotonic calibration on held-out seeds, we expect no student gain, and the proposal reduces to "SLaM works on graphs." That would still be a useful but smaller result.
Why not just build a better teacher, for example with the KNN graph from SFGL or with more propagation steps? A better teacher helps every distillation method equally, and we use the KNN-graph teacher for all of them. The question here is orthogonal: at any teacher quality, how much of the remaining error can the student avoid copying? We test this by repeating the comparison with a weak and a strong teacher and checking whether the gain persists.
Proposed Experiments
We use the citation datasets Cora, Citeseer, and PubMed at 1, 3, and 5 labels per class, plus the 60-label PubMed setting from SFGL. We add ogbn-arxiv at 0.03% and 1% labels, and ogbn-products and Amazon-Computer/Photo at low label rates, drawn from OGB and the GLNN suite. For inductive and cold-start evaluation we use Cold Brew's isolation protocol, in which test-node edges are removed, together with ABIDE and TADPOLE. Penn94 serves as a heterophilic stress test.
Metrics are student accuracy (transductive, inductive, and on isolated nodes, stratified by propagation reach), expected calibration error, teacher-error AUROC for each reliability estimator, and inference latency. Latency should be identical to GLNN's, since the student architecture is the same.
Three checks decide the outcome. The first is whether propagated reliability beats margin-only calibration on error-prediction AUROC. The second is whether that advantage translates into student accuracy, concentrated on low-reach nodes, beyond the SLaM and filtering baselines. The third is whether the advantage disappears when reliability is propagated over a shuffled graph. A positive result on all three supports the claim that the graph should decide how much the student trusts the teacher. A failure on the first check ends the idea cleanly. We expect the method to lose its edge on heterophilic graphs, and we will report that boundary.
References
- Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation
- GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference
- Teaching Semi-Supervised Classifier via Generalized Distillation
- SLaM: Student-Label Mixing for Distillation with Unlabeled Examples
- Label Propagation with Weak Supervision
- On the Equivalence of Decoupled Graph Convolution Network and Label Propagation
- Scale-Free Graph-Language Models
- Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
- SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP
- LaplaceNet: A Hybrid Graph-Energy Neural Network for Deep Semisupervised Classification
- Open Graph Benchmark: Datasets for Machine Learning on Graphs