Many graph models now begin by turning each node's text into a vector with a pretrained language model. The text might be a paper abstract, a product description, or a concept definition. A graph neural network (GNN), which updates each node's vector by mixing in its neighbors' vectors, then trains on top of those starting vectors. This "semantic node initialization" is behind the strongest text-attributed graph results. One example is SimTeG, which fine-tunes a text encoder on node text alone and reaches 77.48% accuracy on OGBN-Arxiv with a plain GraphSAGE. The encoders behind these results were pretrained on web crawls that very plausibly contain the arXiv abstracts, product pages, and Wikipedia entries the benchmarks are built from. Nobody has measured how much of the reported accuracy comes from that exposure.
This proposal argues that on graphs, contamination should be treated as a property of a node's neighborhood, not of the node alone. Message passing mixes a test node's vector with its neighbors' vectors. If the encoder memorized the neighbors' text together with their labels, a test node whose own text was never seen could still be classified correctly for leaked reasons. Standard audits run membership inference on each test item's own text and would miss this. We will build a controlled pretraining study that injects benchmark text at known doses and positions in the graph. With it we measure two quantities separately: exploitation of a test node's own contamination, and exploitation that arrives through its edges. We then test whether a neighborhood-aware audit predicts the inflation in public encoders better than per-node detection does.
Core Claim
We will test two hypotheses. The first is that, in text-initialized GNNs, contamination of a node's neighbors measurably raises accuracy on that node even when the node itself is clean. The second is that this "edge-borne" exploitation grows with the number of graph-mixing layers, so a GNN can exploit memorized text more than an MLP that uses the same features. If both hold, two consequences follow. Per-node membership inference undercounts leakage in graph benchmarks, and an audit has to aggregate detection scores over neighborhoods. If the first hypothesis fails, meaning edge-borne exploitation stays near zero even at heavy duplication, then existing per-node tools are enough and the community can stop worrying about this channel. That is also worth knowing.
Where This Sits
The flat-text version of this question has a clean answer. Data Contamination: From Memorization to Exploitation pretrained BERT on Wikipedia with injected SST and SNLI examples. It separated memorization, measured as label recovery before fine-tuning, from exploitation, measured as the accuracy gap between seen and unseen test items after fine-tuning. It found about 15% memorization but under 1% exploitation for single exposures, rising to 40% exploitation at 200 duplicates, with early-pretraining exposure doing more damage. We reuse that method nearly unchanged. The new element is that the downstream model is a GNN. Every test node's prediction draws on many other nodes' texts, so the "seen versus unseen" split is no longer well defined per item.
Two graph papers motivate the concern without testing it. SimTeG shows that most of the gain on text-attributed graphs comes from the language-model feature stage. It also shows that fine-tuning can pull same-class nodes into tight clusters. That is exactly the behavior one would expect if fine-tuning surfaces memorized labels. Enhancing Future Link Prediction in Quantum Computing Semantic Networks through LLM-Initiated Node Features uses a chronological split, training on 2007–2021 and testing on 2023–2024. Its LLM feature generators, however, were trained on text from that test period. Its isolated-node AUROC of 81–85, against 53.28 for node2vec, may therefore partly reflect recall rather than extrapolation. Neither paper runs a contamination audit.
On the detection side, Min-K%++ is the reference-free membership test we build on. It flags a text as likely training data when its token log-probabilities sit unusually high relative to the vocabulary distribution. It has only been applied to standalone sequences. DyVal uses graphs to generate fresh reasoning problems for language models, but it never evaluates GNNs or node features. It shows that graph structure can be used to defeat memorization, whereas we ask whether graph structure spreads it. Finally, the contamination survey distinguishes data-level contamination (inputs seen) from label-level contamination (input–label pairs seen). We inject both levels separately because they should behave differently on graphs.
Research Program
The controlled pretraining study. We pretrain BERT-base-scale encoders from scratch on a Wikipedia corpus of the size used by Magar and Schwartz, mixed with a fixed pool of arXiv-style abstracts so that the domain is not foreign. Into this corpus we inject OGBN-Arxiv test-node text in two forms. The data-level form is the abstract alone. The label-level form is the abstract followed by its subject category written as a sentence. We vary duplication at 1, 10, and 100 copies, and we vary timing between early and late in pretraining. These choices follow the dose and timing effects the prior work found.
The key manipulation is where in the graph the contamination sits. Test nodes are randomly assigned to one of three conditions, with the assignment stratified by degree. In the first, a node's own text is injected. In the second, the text of some of its test-set neighbors is injected but its own text is not. In the third, neither its own text nor its neighbors' text is injected. Random assignment makes the comparisons causal. The design has three injection forms (data-level, label-level, and a matched no-injection corpus), crossed with three duplication levels and two timings. It needs roughly a dozen pretraining runs, which is the main cost and is comparable to the original flat-text study.
Downstream pipelines. Each encoder feeds the SimTeG two-stage pipeline. The first stage is low-rank adapter fine-tuning on training-node text only. The second stage is frozen features passed to an MLP, a 2-layer GraphSAGE, and a deeper RevGAT. We also run a frozen variant with no fine-tuning, to see whether task fine-tuning is what surfaces memorized labels.
We define self-exploitation as accuracy on self-contaminated test nodes minus accuracy on fully clean ones. We define edge-borne exploitation as accuracy on nodes that are clean themselves but have contaminated neighbors, minus accuracy on fully clean nodes. We report edge-borne exploitation as a function of the number of contaminated neighbors and the number of GNN layers.
A link-prediction replicate. We repeat a reduced grid on the academic citation graph in TEG-DB. There the edge text consists of citation sentences, which adds a third channel: contamination of the edge itself.
The neighborhood audit. For any public encoder that exposes token probabilities, we compute the Min-K%++ score of every test node's text. We then form a neighborhood score that weights neighbors' scores by the GNN's own propagation pattern. In the controlled encoders, ground truth is known, so we can measure how well each score identifies nodes whose predictions are inflated. On public encoders applied to OGBN-Arxiv and to the quantum semantic network, there is no ground truth. There we test whether high neighborhood scores predict nodes that the GNN gets right but the clean-encoder reference gets wrong. We present this as correlational evidence only.
Baselines
The first baseline is the reported score with no audit. On OGBN-Arxiv, SimTeG reports 77.48% accuracy with GraphSAGE and 78.04% with RevGAT plus LLM-generated summaries. On the quantum network, the LLM-initiated features paper reports an MLP AUROC of 87.02. We do not aim to "beat" these numbers. The question is how much of each would remain under a matched clean encoder. Within the controlled study, the corresponding reference is the no-injection encoder trained on an identical token budget, and every exploitation number is measured against it.
The strongest existing method for the auditing question is per-node Min-K%++. It reports AUROC gains of 6.2 to 10.5 points over Min-K% on WikiMIA. We found no reported result on Lacuna for Min-K%++ applied to graph node text. We run it with the same encoder, the same tokens, and the same choice of k tuned on a held-out set of injected training nodes, so the per-node and neighborhood scores get an identical tuning budget. The neighborhood audit beats it if it achieves a clearly higher AUROC at identifying inflated test nodes in the controlled study, measured over the clean-self, contaminated-neighbor nodes where the per-node score has no signal by construction. It must also not lose AUROC on the self-contaminated nodes.
The ablation that isolates the key ingredient compares the same frozen features under an MLP and under GNNs of increasing depth. If edge-borne exploitation is zero for the MLP and grows with depth, the graph-mixing step is responsible. A second ablation swaps label-level injection for data-level injection. If edge-borne exploitation appears only at label level, the mechanism is leaked labels moving through neighbors, not familiarity with the text.
Why not just…?
Why not just run Min-K%++ on test nodes and drop the flagged ones? By construction, this cannot see the edge-borne channel. Its target is the node's own text, while our hypothesized leak comes from neighbors. Whether that omission matters is exactly what the controlled study measures. If edge-borne exploitation is negligible, this simpler approach wins, and we will say so.
Why not just use a temporal split with an encoder whose cutoff predates the test period? This is the strongest alternative, and where it is feasible it is the cleanest fix. Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? shows how sharply models drop on post-cutoff data. It has three limits for graphs. Fixed benchmarks like OGBN-Arxiv cannot be re-split without breaking comparability with past results. Older encoders are weaker, so the accuracy drop mixes contamination with lost capability. And the quantum-network paper shows that chronological graph splits are routinely paired with newer encoders anyway. Our controlled study supplies what a temporal split cannot: dose-response curves that say how much inflation to expect at a given exposure level. We will also run a temporal check on the quantum network with an older open encoder to see whether it agrees with our audit's ranking of nodes.
Why not just use shallow or frozen generic embeddings? Shallow features lose substantial accuracy. Gloss averaging on WordNet (Leveraging Lexical Resources) and node2vec on cold-start nodes both show this. More importantly, a frozen generic encoder is still pretrained on the web. Our frozen-encoder arm tests directly whether avoiding fine-tuning removes exploitation or only reduces it.
Why not assume the flat-text result transfers and stop? The Magar and Schwartz finding of under 1% exploitation at single exposure might carry over. But arXiv abstracts are widely mirrored online, so the duplicated regime, where they found 40%, is the realistic one. Mixing neighbors is also a new pathway that their setting lacked. The MLP-versus-GNN comparison tests whether the transfer assumption holds.
Decisive Checks
Three measurements settle the matter. The first is whether edge-borne exploitation on OGBN-Arxiv exceeds the seed-to-seed noise of the clean encoder at 10 or more duplicates. The second is whether it grows from MLP to GraphSAGE to RevGAT. The third is whether neighborhood scores beat per-node Min-K%++ on the contaminated-neighbor nodes.
A null on the first check falsifies the core claim. A positive on all three would mean that text-attributed graph leaderboards need neighborhood-level contamination audits. In that case, the public-encoder analyses on OGBN-Arxiv and the quantum network give a first, explicitly correlational estimate of how much current reported numbers are affected.
References
- SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning
- Data Contamination: From Memorization to Exploitation
- Enhancing Future Link Prediction in Quantum Computing Semantic Networks through LLM-Initiated Node Features
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Benchmark Data Contamination of Large Language Models: A Survey
- TEG-DB: A Comprehensive Dataset and Benchmark of Textual-Edge Graphs
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Leveraging Lexical Resources for Learning Entity Embeddings in Multi-Relational Data