A Dataset for Distilling Knowledge Priors from Literature for Therapeutic Design

Haydn Thomas Jones, Natalie Maus, Josh magnus Ludan, Maggie Ziyu Huan, Jiaming Liang, Marcelo Der Torossian Torres, Jiatao Liang, Zachary Ives, Yoseph Barash, Cesar de la Fuente-Nunez, Jacob R. Gardner, Mark Yatskar

Abstract

This paper introduces Medex, a comprehensive dataset comprising 32.3 million natural language facts regarding 2 million medical entities, including small molecules and proteins, extracted from scientific literature. By pretraining compact models (as small as 15M parameters) on these "knowledge priors," the authors demonstrate performance that exceeds generalist models 100 times their size on drug discovery benchmarks. This approach facilitates the generation of safer and more biologically viable therapeutic candidates by incorporating experimental knowledge directly into the model's training.

Introduction: The Challenge of Biological Realism

Current AI-driven drug discovery predominantly utilizes in silico simulations to optimize for specific objectives, such as binding affinity. However, these models often "hallucinate" molecules that, while chemically plausible, are biologically unviable due to toxicity, poor absorption, or instability.

An analysis of top-performing models on the GuacaMol benchmark reveals a startling reality: over 60% of proposed molecules possess a high probability of mutagenicity (causing genetic mutations). This deficiency stems from a lack of "experimental priors"—the intuitive biological knowledge human researchers possess that is not captured in standard chemical representations. Much of this critical information remains inaccessible within millions of unstructured scientific papers and patents.

Analysis of molecules proposed by existing models shows that the vast majority are unsuitable due to toxicity or poor ADME profiles.

Methodology: The Medex Pipeline

The researchers developed an automated pipeline to convert unstructured literature into a structured dataset of (entity, fact) pairs, enabling models to learn from the collective experience of the scientific community.

  1. Document Gathering: 3.2 million relevant papers were identified by cross-referencing databases like PubChem and UniProt.
  2. Entity Extraction: Utilizing a distillation approach, a Llama 3.1 405B model identified entities in text, which then served to fine-tune a Llama 8B model for large-scale processing of 400 million paragraphs.
  3. Fact Generation: GPT-4 was employed to summarize paragraphs into concise, "fair-use" facts (e.g., "Levofloxacin is detectable in blood and brain"), which were then distilled into a Gemma 4B model for scale.
  4. MedexCLIP: A contrastive learning model was developed to align chemical structures (SMILES strings) with their corresponding natural language facts in a shared embedding space.
The Medex pipeline transforms unstructured scientific literature into a structured dataset of 32.3 million fact pairs across three stages: processing, extraction, and generation.

Results and Key Insights

The integration of literature as a prior significantly enhances model performance and safety, proving that high-quality, domain-specific data can compensate for smaller model scales.

  • Extreme Parameter Efficiency: A 15M parameter MedexCLIP model outperformed the 2B parameter TxGemma on 23 of 28 regression tasks in the Therapeutic Data Commons (TDC) benchmark.
  • Zero-Shot Reasoning: Because the model learns to associate structures with descriptive text, it can perform zero-shot classification. MedexCLIP achieved a 74% relative improvement over a base Gemma 2B model in zero-shot evaluations across nine safety and ADME (Absorption, Distribution, Metabolism, and Excretion) endpoints.
  • Safer Molecular Design: When applied as a constraint in molecular optimization, Medex-informed models proposed candidates with significantly lower toxicity while maintaining high effectiveness compared to unconstrained models.

Discussion and Future Work

While Medex represents a significant advancement, the authors acknowledge several limitations. The current pipeline treats documents independently, without cross-referencing to resolve contradictions or weight sources. Furthermore, the dataset inherits publication biases inherent in scientific literature, such as the over-reporting of positive results. Future work will focus on expanding the dataset and developing knowledge graph structures to enable more complex multi-hop reasoning between extracted facts.

Summary updated

Abstract

AI-driven discovery can greatly reduce design time and enhance new therapeutics' effectiveness. Models using simulators explore broad design spaces but risk violating implicit constraints due to a lack of experimental priors. For example, in a new analysis across diverse models on the GuacaMol benchmark using supervised classifiers, over 60\% of molecules proposed had a high probability of being mutagenic. In this work, we introduce Medex, a dataset of priors for design problems extracted from literature describing compounds used in lab settings. It is constructed with LLM pipelines for discovering therapeutic entities in relevant paragraphs and summarizing information in concise fair-use facts. Medex consists of 32.3 million pairs of natural language facts, and appropriate entity representations (i.e. SMILES or RefSeq IDs). To demonstrate the potential of the data, we train LLM, CLIP, and LLaVA architectures to reason jointly about text and design targets and evaluate on tasks from the Therapeutic Data Commons (TDC). Medex is highly effective for creating models with strong priors: in supervised prediction problems that use our data for pretraining, our best models with 15M learnable parameters outperform larger 2B TxGemma on both regression and classification TDC tasks, and perform comparably to 9B models on average. Models built with Medex can be used as constraints while optimizing for novel molecules in GuacaMol, resulting in proposals that are safer and nearly as effective. We release our dataset on HuggingFace at https://huggingface.co/datasets/DocAndDesign/Medex, and will provide expanded versions as the available literature grows.
BibTeX
@misc{jones2025dataset,
  author = {Haydn Thomas Jones and Natalie Maus and Josh magnus Ludan and Maggie Ziyu Huan and Jiaming Liang and Marcelo Der Torossian Torres and Jiatao Liang and Zachary Ives and Yoseph Barash and Cesar de la Fuente-Nunez and Jacob R. Gardner and Mark Yatskar},
  title = {{A Dataset for Distilling Knowledge Priors from Literature for Therapeutic Design}},
  year = {2025},
  eprint = {2508.10899},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10899},
}
Figure