Language models trained on specialist text, such as biomedical abstracts, can often recall single facts but fail when an answer requires chaining two of them ("which proteins are targeted by drugs used for disease X?"). A recent remedy builds a graph of entities across a document collection, walks paths through it, and has a large model write synthetic training text along those paths, so that the student model sees cross-document links during continued pre-training. Continued pre-training means further training an existing model on raw text from a new domain. The weak point is how the graph gets its edges. Two entities are linked whenever they appear in the same passage. Decades of relation-extraction work show that co-occurrence is often coincidental, so a large share of these links assert nothing. Synthetic text written along such paths teaches the model associations that are fluent and confident but false.
We propose to verify each edge before it is used. An edge survives only if at least one sentence in the corpus actually states a relation between its two entities, and edges through overly connected "hub" entities are down-weighted. Synthetic reasoning text is then written only from the asserting sentences, not from whole surrounding paragraphs. The central claim is testable: on biomedical two-hop benchmarks, a model continued-pre-trained on synthesis from verified edges beats a model trained on the same number of tokens from co-occurrence edges. It should also produce fewer answers whose only support is a co-occurrence shortcut. The gain should grow with the level of edge noise, and we will measure that dependence directly by injecting noise under control.
Where this sits
The method we extend is Synthesize-on-Graph (SoG). SoG builds a cross-document context graph from entity co-occurrence and samples paths by breadth-first search guided by paragraph similarity. It then generates chain-of-thought and contrastive text for continued pre-training, and reports gains over EntiGraph on MultiHop-RAG and BIOASQ. SoG does not check whether an edge expresses a relation, and its authors note that training becomes unstable with synthetic paths. We keep SoG's synthesis step and change only which edges and which evidence text reach it. That isolates edge validity as the variable under study.
Our motivation comes from two separate findings. Relation-extraction work shows that distant supervision and co-occurrence yield many false positives (Filtering Relational Noise in Information Extraction; Intxaurrondo et al.). Work on multi-hop retrieval shows the same problem in a graph setting. PAGE-RAG names it the "connectivity-support gap": a shared hub entity links passages without making one support the other. PAGE-RAG addresses this at test time, as a per-query reranker. It penalizes hubs and scores edge provenance, and reports Support F1 +10.4 and Answer F1 +3.3 averaged over HotpotQA, MuSiQue and 2WikiMultiHopQA. Being a reranker, it cannot recover evidence the retriever never returned, and it leaves the model's parameters unchanged. We move the same idea into the training data.
Our edge verifier borrows its structure from multiple-instance learning for document-level relation extraction. Multiple-instance learning treats a group of examples as a "bag" that is labeled positive if any one member is positive. KMGRE uses this setup, with an entity pair as the bag and its mention pairs as instances. It alternates between guessing which few mention pairs are key and retraining on those, and reports 76.71% F1 on DWIE. KMGRE's own cost figures also shape our design: it needs two pre-trained models, with 22% more training time and 63% more GPU memory than ATLOP. We therefore use a single small encoder and run it once, offline, over the corpus.
Hyperlink-based retriever pre-training (LOUVRE, PathR@1 67.0 on HotpotQA) shows that path-structured pre-training data helps. It depends, however, on curated links that biomedical abstracts lack, which is exactly why we have to build and clean edges from raw text. Finally, Männistö et al. find that what a model saw during pre-training predicts task performance better than the choice of fine-tuning method. That finding is our reason to intervene at pre-training rather than only at inference.
Research Program
The corpus is PubMed abstracts, the same source used by BioBART, and entities are tagged with UMLS types. For every co-occurring entity pair, we collect all sentences mentioning both entities. A sentence-level assertion scorer then estimates whether each sentence states a relation between the pair or merely mentions both. The scorer is a biomedical encoder, trained by distilling from a strong model's judgments on a labeled sample of roughly 20k sentence–pair items and refined with the key-mention procedure from KMGRE. The edge score is the maximum over the pair's sentences, following the rule that one asserting sentence suffices.
We multiply that score by an inverse-degree specificity term, $s(u,v)=a(u,v)\cdot \log(N/d_u)\log(N/d_v)/\log^2 N$. Here $a(u,v)$ is the assertion score, $d_u$ is the number of distinct neighbors of entity $u$, and $N$ is the number of entities. The term is borrowed from inverse document frequency weighting, and it pushes hubs like "inflammation" toward zero without removing them outright.
Paths are sampled by SoG's procedure, restricted to edges whose score exceeds a threshold. For each sampled path, the synthesizer receives only the top asserting sentence for each edge, together with its PubMed identifier. This keeps the generated chain-of-thought tied to text that actually states each hop. Everything else is held fixed across conditions: the synthesis prompts, the student model (Qwen2.5-3B-Instruct, as in SoG), and the synthetic token budget.
The study has three parts. First, we measure edge-level noise directly. PrimeKG serves only as an external audit: we use it to estimate the precision of co-occurrence edges and of verified edges on the entity types it covers. It is never used to build training data, since BioHopR is derived from it. Second, we run the main comparison of continued pre-training variants. Third, we run a controlled noise-injection study. Starting from the verified graph, we add random co-occurrence-only edges at rates of 0%, 25%, 50% and 75%. We then trace how downstream two-hop accuracy and the spurious-bridge rate, defined in the experiments section, change with the injection rate.
Baselines
The naive reference points are Qwen2.5-3B-Instruct with no further training, and the same model continued-pre-trained on raw PubMed text at the matched token count. These show how much any graph-guided synthesis adds over plain domain exposure.
The strongest existing competitor for training the model is SoG itself, reimplemented on our corpus with its co-occurrence graph, BFS sampling, and chain-of-thought plus contrastive synthesis. It is the only method in the file that continues pre-training on synthesized cross-document paths, so it is the direct test of our change. The file gives no reported SoG number on MultiHop-RAG or BIOASQ: no reported result found on Lacuna for SoG scalars on those benchmarks. We therefore compare against our own replication. The replication is run under the same synthetic token budget, the same synthesizer model, the same learning rate schedule tuned once on a held-out perplexity split, and three seeds.
Two ablations isolate our key ingredient. The random-drop ablation removes the same fraction of edges our filter removes, but at random. It separates "cleaner edges" from "smaller, sparser graph." The hub-only ablation keeps the specificity term and drops the assertion score, which separates hub suppression from assertion checking.
For reference points on the evaluation benchmarks, we use the numbers the file reports. On BioHopR, o3-mini reaches 37.93% precision on 1-hop and 14.57% on 2-hop, and open-source biomedical models score near zero on 2-hop. On MedHopQA, DeepRAG reports 62.4% exact match and 71.8% concept accuracy, against 54.3% and 66.5% for standalone DeepSeek R1. DeepRAG uses a far larger reasoning model with preference-tuned retrieval, so we treat it as a ceiling to report against rather than as a matched comparison.
We count it as beating the strongest baseline if verified-edge training beats SoG on BioHopR 2-hop precision and MedHopQA exact match, beyond seed variance and in both closed-book and retrieval-augmented modes. The spurious-bridge rate must also be lower. A 3B model that approaches o3-mini's 14.57% two-hop precision would be a notable secondary result, but it is not our success criterion.
Why not just…?
Why not just filter SoG's generated passages afterward with a language-model judge? This is the cheapest competitor and a real one. By the time a passage is written, however, the synthesizer has already been given whole paragraphs around coincidental co-mentions. A judge can reject a passage that is obviously wrong, but it cannot tell whether a fluent bridge claim was actually stated in the source. We include this variant, with the same judge model and the same retained token count. If it matches our method, the conclusion is that where the filtering happens does not matter, and we will report it that way.
Why not just ask a large model whether each edge is valid, instead of training an assertion scorer? It may well work as well. Our claim concerns edge validity, not the particular verifier. We therefore run the language-model edge judge as a variant on a subsample, and on the full graph if the budget allows. We compare its edge precision (audited against PrimeKG) and its downstream accuracy with the distilled scorer. If the two tie, the distilled scorer's contribution is cost alone, since it makes a single pass over millions of pairs, and we will say so.
Why not skip pre-training and use PAGE-RAG at inference with the base model? PAGE-RAG's own analysis says it cannot recover hops that are missing from the candidate pool. It also leaves the model unable to recognize a bridge entity it has never seen linked. To settle the question, we run base model + PAGE-RAG, our model + plain top-$k$ retrieval, and our model + PAGE-RAG, all under a fixed context budget. If base + PAGE-RAG matches our model + plain top-$k$ retrieval, training-time denoising adds nothing beyond test-time denoising. If the gains add up, the two methods are complementary.
Why not just fine-tune on multi-hop QA pairs with LoRA? BioHopR reports that medically fine-tuned open models remain near zero on 2-hop questions. Männistö et al. find that pre-training matters more than the choice of fine-tuning method. We add a LoRA baseline trained on MedHopQA training questions for completeness.
Proposed Experiments
Evaluation covers BioHopR at 1-hop and 2-hop, with precision and recall under its BioLORD-2023-C matching threshold $\tau=0.9$. It also covers MedHopQA exact match and concept accuracy, and BIOASQ. MultiHop-RAG, a news benchmark, is a check that the method does not overfit to biomedicine. Each model is tested closed-book and with retrieval over the same PubMed index. We also track the 1-hop to 2-hop drop as a direct measure of the wall BioHopR describes.
We add a diagnostic metric, the spurious-bridge rate. For wrong 2-hop answers, it is the fraction where the model's stated bridge entity co-occurs with both endpoints in the corpus but has no asserting sentence and no PrimeKG edge. This is the failure our method predicts SoG will produce more often.
Two checks decide the claim. The first is the noise-injection curve. If our explanation is right, SoG-style training should degrade steadily as spurious edges are added, and the spurious-bridge rate should rise in step. If accuracy stays flat, edge noise is not what limits these models and the proposal is wrong. The second check is the three-way comparison of verified edges, random drop, and hub-only at matched tokens. If random drop performs as well as verified edges, the benefit comes from sparsity rather than validity.
Secondary analyses report training stability, measured as loss spikes and degraded general ability on single-hop QA, to test whether cleaner paths reduce the instability SoG's authors observed. They also report sensitivity to the edge-score threshold, which trades graph coverage against precision.
References
- Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continued Pre-training of Large Language Models
- Learning to Extract Relations from the Web using Minimal Supervision
- UBC at Slot Filling TAC-KBP 2011
- PAGE-RAG: Provenance-Aware Graph Evidence Promotion for Fixed-Budget Multi-hop Retrieval-Augmented Generation
- Key Mention Pairs Guided Document-Level Relation Extraction
- Weakly Supervised Pre-Training for Multi-Hop Retriever
- A Comparative Study of PEFT Methods for Python Code Generation
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain
- DeepRAG: Integrating Hierarchical Reasoning and Process Supervision for Biomedical Multi-Hop QA