A neural network with frozen random weights, where only a linear classifier is trained on top, is often a surprisingly strong model. On the PROTEINS graph benchmark such a network beats most published self-supervised graph methods. The idea here is to use that cheap, exactly reproducible control as a common reference point for the whole literature. Almost any published number can be re-expressed as "gain over the same architecture with random weights." We can also rerun the published methods ourselves under matched tuning budgets. That gives two samples: the gains the literature reports, and the gains that unselected, fairly tuned runs actually produce. The gap between the two, fitted with a statistical model of selective reporting, estimates how much of a field's reported progress is manufactured by tuning asymmetry, seed selection, and the file drawer. The file drawer here means experiments that were run but never published because they came out negative.
This matters because the evidence so far is anecdotal in a specific way. Graph Contrastive Learning versus Untrained Baselines: The Role of Dataset Size showed that an untrained 3-layer Graph Isomorphism Network (a standard graph neural network, here left at its random initialization) reaches 74.61% accuracy on PROTEINS. It also showed that every graph contrastive learning method it tested lost to a random network or to simple graph statistics on at least one dataset. What it did not do is say how large the reporting distortion is, or where it comes from. We propose to turn the untrained network from a one-off embarrassment into a measuring instrument.
Core Claim
Publication bias in architecture-driven machine learning can be estimated quantitatively. The key object is a "paired random-weight delta": the trained model's score minus the score of an identical architecture with frozen random weights and an identical linear probe, both tuned under the same budget. This delta is the right unit of analysis because it cancels the probe, the data split, and the architecture's built-in structural bias. What remains is what optimization of the weights actually contributed.
Let $\Delta(n)$ be this true delta at training-set size $n$, measured from our own symmetric reruns. Let $\hat\Delta_i$ be the delta implied by published result $i$, meaning the published trained score minus our reproduced untrained score on the same split. We model
$$\hat\Delta_i = \Delta(n_i) + b_{\text{tune}} + b_{\text{seed}} + \varepsilon_i, \quad \text{observed only if } \hat\Delta_i > c_i.$$
Here $b_{\text{tune}}$ is the inflation from tuning the new method harder than its baselines, and $b_{\text{seed}}$ is the inflation from reporting a favorable subset of random seeds. The noise $\varepsilon_i$ comes from split and seed variance, and $c_i$ is the unknown threshold a result must clear to be published.
This extends the truncated observation model of Unraveling overoptimism and publication bias in ML-driven science in two ways. That work fits learning curves and truncation thresholds to published medical-classification accuracies alone. It treats the model as a black box and must identify everything from the selected sample. We change the quantity being modeled from raw accuracy to gain over an untrained structural prior. We also add an unselected sample: our reruns. With that second sample, the truncation threshold and the bias terms become far better identified than they are from published data alone.
Research Program
The first component is a controlled rerun suite. It starts in graph-level classification, the regime where Graph Contrastive Learning versus Untrained Baselines supplies the method and the motivating evidence. For each trained method (GraphCL, JOAO, SimGRACE, InfoGraph) we pair the identical encoder at random initialization. We evaluate both with the same frozen-embedding linear probe on the TU datasets and on nested subsamples of OGBG-MOLHIV. Every arm gets the same hyperparameter search budget, counted in trials and GPU-hours, over a search space that covers both the encoder and the probe.
The untrained arm needs particular care. Choosing among random weight distributions is itself a form of tuning, so that choice is charged against the untrained arm's budget. The distributions we search over are Gaussian, orthogonal, and the per-pass random diagonal operators of RAP-GNN from On the Effectiveness of Random Weights in Graph Neural Networks. Every run logs all seeds, and nothing is dropped. The output is the curve $\Delta(n)$ with full seed distributions, reported both at the default budget and at the budget each original paper states it used.
The second component is a corpus of published results. We extract every reported number for these methods and their successors on the same datasets and splits, together with whatever tuning and seed information the papers disclose. Each number is converted to $\hat\Delta_i$ using our reproduced untrained score. This conversion is the step that makes the random-weight control useful beyond one paper. Because the control is cheap and deterministic given its seed and distribution, it can anchor papers that never ran it.
The third component is a decomposition of the gap between published and rerun deltas into the three sources in the model. We estimate $b_{\text{tune}}$ directly by rerunning each method under an asymmetric budget, with the method tuned heavily and the baselines left at defaults, and comparing that to the symmetric budget. This mirrors the reversal experiment of Over-optimistic evaluation and reporting of novel cluster algorithms: an illustrative study. In that study a clustering method's apparent lead (AMI 0.7881 against 0.0007 for default DBSCAN) reversed when DBSCAN was tuned (0.8300). That study used classical clustering; we move the manipulation into deep representation learning and make the probe an explicit tuning target. We estimate $b_{\text{seed}}$ from best-of-$k$ order statistics of the logged seed distributions. Whatever inflation remains after removing both is attributed to truncation, and $c_i$ is fitted by maximum likelihood on the truncated model.
The fourth component is a transfer test to node classification on Cora, CiteSeer, and PubMed, with RAP-GNN as the untrained-structure arm. Gains there are already small in the reported literature, which makes it a sensitive second test.
Baselines
The naive reference point is the handcrafted graph statistics probe from Graph Contrastive Learning versus Untrained Baselines: node count, mean degree, and degree histogram fed to a linear classifier, plus the majority-class rate. Any trained method, and any untrained network, must clear it for the delta to mean anything. It is run on identical splits with the same probe tuning budget.
The strongest untrained competitor is the best random-weight encoder under a budget-charged search over distributions. Its anchor numbers from the file are the 74.61% of the untrained 3-layer network on PROTEINS and RAP-GNN's 82.42% against 81.50% for an end-to-end GCN on Cora (On the Effectiveness of Random Weights in Graph Neural Networks). The strongest trained competitor is GraphCL on OGBG-MOLHIV. Khanna et al. report that it overtakes untrained networks at roughly 4,000 to 4,500 graphs and holds a modest ~2% ROC-AUC advantage at the full 41,000 graphs. No reported result was found on Lacuna for exact per-method GCL accuracies on the other TU datasets, so those are produced by our reruns.
The strongest existing bias-estimation method is the truncated regression of Unraveling overoptimism and publication bias in ML-driven science, fitted to the published corpus alone. No reported result was found on Lacuna for its fitted parameters. We run it on exactly our corpus with the same likelihood and optimizer. Our estimator beats it if it recovers held-out quantities more accurately. The test uses semi-synthetic corpora: we simulate selection with a known threshold and known tuning and seed bias from our own logged reruns, and measure error in recovering them.
Two ablations isolate the key ingredient. The first drops the rerun anchor and fits on published raw accuracy instead of the random-weight delta, which tests whether the pairing matters. The second replaces symmetric budgets with each paper's stated budget, which tests whether matched tuning matters.
Why not just…?
Why not just add untrained baselines to more benchmarks, as Khanna et al. recommend? That is necessary, and our rerun suite does it. But it only answers whether a gap exists. It cannot say how much of the literature's reported gain is inflation or where that inflation comes from, because it never compares against the published record. The concrete difference between the two approaches is the decomposition into $b_{\text{tune}}$, $b_{\text{seed}}$, and truncation.
Why not just equalize tuning budgets and rerun? This is the real threat. If symmetric reruns reproduce published deltas within seed noise, the modeling component adds little, and the honest conclusion is that tuning asymmetry is the whole story or that there is no story. Our asymmetric-versus-symmetric experiment measures $b_{\text{tune}}$ directly. If it accounts for the full published-minus-rerun gap, we report that as the finding and the truncation estimate is dropped. The reason to expect a residual is that tuning cannot explain results that were never published. The Rings failure in the clustering study is one example. Another is the tension between static and continual settings: The Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learning reports untrained encoders beating trained continual methods by +23.6% on CoraFull and +38.5% on Roman Empire, yet that literature keeps centering complex trained methods.
Why not just count how many papers omit an untrained control? The count measures how often the control is missing, not how much the omission distorts results. A field in which 90% of papers omit the control but trained gains are real is healthy. Our conversion of published numbers into deltas yields the count as a byproduct, and we will report it alongside the magnitude estimate.
Why not just fit the existing truncation model to published accuracies? It must identify the threshold from the selected sample alone, and it cannot separate a genuinely better method from a harder-tuned one. The anchor-free ablation above tests this directly on semi-synthetic corpora with known ground truth.
Proposed Experiments and Decisive Checks
The primary experiment produces $\Delta(n)$ on OGBG-MOLHIV subsamples (scaffold split, ROC-AUC) and on the TU datasets (10-fold accuracy), with all seeds logged. It is implemented in THUDM/CogDL, which provides the self-supervised graph operators. Alongside it we build the published corpus converted to $\hat\Delta_i$.
The first decisive check is whether published deltas at small $n$ sit systematically above the symmetric rerun distribution, beyond what best-of-$k$ seed selection explains. A clear "no" would be a publishable negative result about the file drawer itself. The second check tests the specific open question from the file: whether tuning only the linear probe on random features, at the budget given to the trained encoder, closes the ~2% ROC-AUC gap at full MOLHIV scale. The third check is the semi-synthetic recovery test, which settles whether the paired anchor improves identification of the truncation threshold over the published-only model. Finally, if the fitted inflation transfers to node classification (Cora, PubMed), the method generalizes. If it does not, the finding is specific to small graph-level benchmarks, and we report that boundary.
References
- Graph Contrastive Learning versus Untrained Baselines: The Role of Dataset Size
- Unraveling overoptimism and publication bias in ML-driven science
- On the Effectiveness of Random Weights in Graph Neural Networks
- Over-optimistic evaluation and reporting of novel cluster algorithms: an illustrative study
- The Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learning