Choosing the Graph Architecture for Scientific Knowledge Extraction Before the Graph Exists

Tools that turn piles of scientific papers into knowledge graphs, meaning networks of concepts linked by relations such as "is a subfield of" or "is used for", increasingly end with a small graph neural network. A graph neural network (GNN) is a model that updates each node by passing messages along its edges. Graph learning has a well-documented weakness: the best GNN design changes from one dataset to the next, and the winner on one graph seldom transfers to another. The fixes that exist all describe a dataset by looking at its graph. That cannot work for literature extraction, because the graph is what we are trying to produce. Before extraction, all we have is the text, and the clearest summary of a field's text is the geometry of its domain-specific word embeddings.

This note proposes to test whether that geometry can stand in for the missing graph. The claim is that statistics computed from a domain-adapted encoder's term vectors over a new corpus can predict which message-passing design will extract that field's knowledge graph best. Such statistics include how tightly hypernym directions align, how dense the neighborhoods of technical terms are, and how often terms get split into word pieces. The prediction would work with no labeled graph and only a few trial trainings. The first step is to measure whether the problem is real. If extraction quality barely changes across architectures within computer science, physics, biology and materials science, the project should stop there and report that. If it does change, we would have the first way to pick an extraction architecture for an unseen field. It would rely only on text, which is the one thing every literature-synthesis pipeline already has.

Core Claim

Three bodies of work frame the gap. On the motivating side, NAS-Bench-Graph evaluated 26,206 GNN architectures on 9 datasets and found that "transferability is a trap": top architectures rarely carry over between datasets, and random search is a hard baseline to beat. The methods that try to transfer design knowledge instead describe tasks numerically from the graph itself. AutoTransfer uses curvature summaries of randomly initialized anchor GNNs and matches tasks against a bank of 120,000 evaluated combinations. KEGNAS encodes graph topology with a variational graph autoencoder to warm-start evolutionary search. AutoGFM routes graphs to operation choices inside a shared supernet, after explicitly discarding domain-specific features as noise. Every one of these needs the input graph, so none of them can be used before the graph exists.

On the application side, HGNet builds hierarchical knowledge graphs from papers on top of SciBERT. It uses a hand-fixed three-channel message-passing design (parent-to-child, child-to-parent, peer-to-peer) and a loss that forces the output to be acyclic. The authors note that this loss cuts real edges when a field's concepts are genuinely circular. HGNet's design is therefore one manual choice applied unchanged across four domains.

On the representation side, work such as Cancer Hallmark Text Classification Using Convolutional Neural Networks and Exploring the effects of drug, disease, and protein dependencies on biomedical named entity recognition shows that domain-trained embeddings encode field-specific structure that general embeddings blur. Its authors use that fact to build better extractors, not to choose architectures.

Our contribution differs from AutoTransfer and KEGNAS in what the task descriptor is made from. We use domain-embedding statistics of the raw corpus, so the method applies before any graph exists and also to fields outside the bank. It differs from HGNet because the message-passing design, including how strictly acyclicity is enforced, becomes a per-field choice that is predicted rather than fixed.

Research Program

The first component is a small, fully evaluated table of extraction architectures. We keep SciBERT frozen and cache its span representations, so each configuration trains only a light graph head over a fixed encoder. This makes exhaustive evaluation affordable, unlike the roughly 8,000 GPU hours NAS-Bench-Graph needed for a from-scratch search space. The search space varies the parts of HGNet's design that plausibly depend on the field. These are which of the three message channels are active, the neighbor aggregator (sum, mean, max or attention), depth and skip-style feature fusion, and the weight on the acyclicity penalty, including setting it to zero. The space contains a few hundred configurations.

The configurations are evaluated on every domain of HGNet's SPHERE corpus (10,000 documents, 111,000 relations), on SciERC, and on ClimateIE from TaxoDrivenKG. Six domains are too few to learn a predictor. We therefore split SPHERE's domains into subfield corpora by topic clustering, aiming for roughly two dozen extraction tasks. Each subfield's held-out relations come from SPHERE's existing annotations.

The second component is the descriptor. For each task's raw text we compute a fixed set of geometric statistics from the domain encoder's term vectors. These include the spread of the vectors in different directions (anisotropy), how consistent the offset is between a candidate parent term and its child terms, local neighborhood density, the word-piece fragmentation rate of technical terms, and the share of sentences in which candidate terms co-occur in both orders, which is a text-side signal of circular relations. A ranking model maps the descriptor to a predicted ordering of configurations. We evaluate it leaving one domain out at a time, holding out whole parent domains rather than single subfields so that close neighbors cannot leak information. The analysis also asks which statistic predicts which design choice. We expect co-occurrence asymmetry to predict whether the acyclicity loss helps, and hypernym-offset consistency to predict whether the directional channels beat the peer channel. That kind of mechanistic link is what would make the result more than a lookup table.

Baselines

The naive reference is HGNet's fixed hand-designed configuration, used on every domain without change. This is what practitioners do now. The file gives HGNet's results only as relative gains (a 5.99% Rel+ F1 improvement on standard relation-extraction benchmarks and 26.20% in zero-shot). No reported result was found on Lacuna for HGNet's absolute Rel+ F1 on SPHERE, so we will reproduce it on our splits.

The strongest search competitor is random search. NAS-Bench-Graph reports it often outperforms reinforcement-learning and evolutionary search, though the file gives no single comparison number. It gets the same frozen encoder, the same search space and the same validation data. We compare at equal trial budgets of 1, 3, 10 and 30 trainings.

The strongest transfer competitor is AutoTransfer's anchor-model descriptor. It is computed on a proxy graph obtained by zero-shot extraction with an open LLM, and it uses the same task bank and nearest-neighbor transfer. AutoTransfer reports a Kendall rank correlation of 0.43 between its task similarity and model rankings, and reports that 3 transferred trials beat 30 trials of cold-start AutoML. Those two numbers are our reference points for rank quality and budget efficiency. A second structural competitor uses plain statistics of that same proxy graph: homophily, cycle rate and hierarchy depth. We also include an AutoGFM-style weight-sharing supernet trained across all training domains, which makes its own per-graph operation choices at inference.

Two ablations isolate the key ingredient. In the first, the same descriptor is computed from a general-purpose encoder instead of SciBERT, which tests whether the domain adaptation is what carries the signal. In the second, the descriptor is computed from a domain-continued-pretraining encoder in the spirit of Optimal Splitting of Language Models from Mixtures to Specialized Domains, which tests whether more specialization helps further. Beating the competitors means two things. First, the configuration chosen with zero or three trials must reach higher held-out Rel+ F1 than random search with ten trials. Second, it must reach a leave-one-domain-out Kendall correlation above the proxy-graph descriptors. The comparison uses the same bank, and wins must hold on a majority of held-out parent domains.

Why not just…?

Why not just run random search for a few trials? This is the most serious threat. If the extraction head is cheap, ten random trials may make any predictor unnecessary. The budget curves above settle the question directly. The predictor is only worth having if it matches the best of ten random trials with zero or one trial, or if it beats random search consistently at low budgets. If random search catches up by three trials, we will say the problem does not need prediction and report the sensitivity table as the main finding.

Why not just extract a rough graph first and compute graph statistics on it? This is also a real threat, and it is why the proxy-graph descriptors are baselines rather than an afterthought. Our reason to doubt it is that zero-shot proxy graphs are noisy exactly where the decision matters. Scientific Knowledge Graph and Ontology Generation using Open LLMs found that experts judged only 64.5% of sampled zero-shot relations correct. HGNet reports that general LLMs hallucinate impossible cycles, which would distort exactly the cycle rate that should decide the acyclicity weight. If proxy statistics nonetheless predict as well as embedding geometry, the text-geometry claim is refuted for this setting.

Why not just train one shared supernet across fields, as AutoGFM does? A supernet that adapts per graph could remove the selection problem entirely. It still routes on the input graph, though, and it treats domain features as noise to strip away. We run it on the same domains. If its held-out Rel+ F1 matches the predicted per-field configuration, the shared supernet is the better engineering choice.

Why not just have a frontier LLM extract the graph directly? HGNet reports that its roughly 300M-parameter model beat Llama-3-70B and GPT-3.5 at relation extraction while running much faster. TaxoDrivenKG needed taxonomy grounding to cut hallucinated entities by 23.3%. We therefore treat direct LLM extraction as an outside reference rather than the main competitor. We also include one cheap variant in which an LLM is shown a description of the domain and asked to choose a configuration, which tests whether the descriptor's information is simply common knowledge.

Proposed Experiments

The first decisive check is sensitivity. On the full table, we measure how much held-out Rel+ F1 separates the best per-domain configuration from HGNet's default, and whether the configuration rankings of different domains disagree. A gap smaller than the variation across random seeds ends the project with a clean negative result. That result would itself contrast with NAS-Bench-Graph's finding that transfer fails in node classification.

The second check is prediction. We report leave-one-domain-out Kendall correlation and Rel+ F1 at each trial budget against every baseline above. We also report ClimateIE as a fully out-of-bank field: no SPHERE subfield resembles it, so it is exactly where AutoTransfer's bank-matching is known to fail.

The third check is mechanism. We run the general versus domain encoder ablation, and we test whether co-occurrence asymmetry in the text predicts the domains where turning off the acyclicity loss raises recall of true cyclic relations without hurting precision. That would address the limitation HGNet itself reports.