Structure Inducing Pre-Training

Matthew B. A. McDermott, Brendan Yap, Peter Szolovits, Marinka Žitnik

Abstract

While language model pre-training has fundamentally shifted the machine learning landscape, the mechanisms by which it improves downstream performance—particularly for whole-sample embeddings—remain poorly understood. This is especially evident when adapting these methods to non-linguistic domains like biology or medicine. We introduce Structure Inducing Pre-Training (SIPT), a framework that explicitly imposes relational structure on latent spaces using external graphs. By decomposing pre-training objectives into intra-sample and structure-inducing components, we provide a theoretical foundation for how relational inductive bias influences fine-tuning performance. Our empirical results across three data modalities and ten tasks demonstrate that SIPT consistently outperforms standard pre-training baselines, offering a roadmap for more intentional transfer learning design.

Introduction

In Natural Language Processing (NLP), pre-training typically focuses on "per-token" objectives, such as predicting a masked word. When these models are applied to "per-sample" tasks—like classifying the sentiment of an entire document—they rely on the model's internal ability to aggregate token-level features into a coherent global embedding. However, in domains such as proteomics or clinical informatics, the primary unit of interest is often the whole sample (e.g., a protein sequence or a patient record).

Current pre-training methods often fail to impose explicit or deep structural constraints on these per-sample latent spaces. While contrastive learning offers some implicit geometric constraints through data augmentation, it lacks a mechanism to reason about the precise distances between disparate samples. SIPT addresses this gap by leveraging domain-specific knowledge, represented as graphs, to shape the geometry of the embedding space.

The SIPT Framework

The core of the SIPT framework is the decomposition of the pre-training loss into two distinct objectives:

  1. Intra-sample Objective ($\mathcal{L}_M$): This represents standard self-supervised tasks, such as Masked Language Modeling (MLM), which focus on the internal structure of a single data point.
  2. Structure-inducing Objective ($\mathcal{L}_{SI}$): This new term regularizes the latent space geometry based on a user-defined Pre-Training Graph ($G_{PT}$).

By utilizing $G_{PT}$, researchers can encode known relationships directly into the model's training signal. For instance, if two proteins are known to interact in a biological pathway, an edge in $G_{PT}$ ensures the $\mathcal{L}_{SI}$ term pushes their embeddings closer together. We implement this using Structure-Preserving Metric Learning (SPML), adapting multi-similarity and contrastive losses to ensure the latent space reflects the connectivity and topology of the input graph.

Theoretical Foundations: Explicit and Deep Constraints

We propose a taxonomy to categorize pre-training constraints based on two dimensions:

  • Explicitness: A constraint is explicit if it allows for direct reasoning about the distance or similarity between any two samples in the dataset.
  • Depth: A constraint is deep if it requires a high-dimensional manifold to be satisfied, preventing the model from simply collapsing data into a few discrete clusters.

Our theoretical analysis culminates in Theorem 1 (The Performance Floor), which establishes a mathematical link between the pre-training graph and downstream success. We prove that if the $G_{PT}$ aligns with the downstream task (a property termed "Local Consistency"), the nearest-neighbor accuracy of the fine-tuned model is lower-bounded by that consistency. Essentially, a well-constructed pre-training graph provides a guaranteed performance baseline.

Experimental Results

We validated SIPT across three distinct modalities and ten fine-tuning tasks, comparing it against standard per-token and per-sample baselines:

  • Proteomics: Using the "Tree-of-life" dataset, we utilized a protein-protein interaction network as $G_{PT}$. SIPT achieved a 13.1% relative reduction in error (RRE) on protein stability tasks.
  • Biomedical Text: Using citation networks from the Microsoft Academic Graph to guide the pre-training of abstract embeddings, SIPT reached an RRE of 17.7% on specific classification tasks.
  • Network Analysis: Using protein-protein interaction ego-graphs, we demonstrated that SIPT creates more meaningful clusters in the latent space compared to standard masked pre-training.

Beyond raw accuracy, SIPT models exhibited faster convergence, reaching peak performance levels significantly earlier in the fine-tuning process than baseline models. Visualizations via UMAP confirmed that SIPT induces a latent space with clearer, more biologically or logically relevant clusters.

Conclusion

SIPT demonstrates that the "black box" of pre-training can be opened and guided by explicit relational structures. While the success of the method depends on the quality and relevance of the pre-training graph, it provides a powerful tool for domains where relational data is abundant but labeled task data is scarce. Future work will focus on automating graph discovery and exploring higher-order topological features to further refine the geometry of induced latent spaces.

Summary updated

Abstract

Language model pre-training and derived methods are incredibly impactful in machine learning. However, there remains considerable uncertainty on exactly why pre-training helps improve performance for fine-tuning tasks. This is especially true when attempting to adapt language-model pre-training to domains outside of natural language. Here, we analyze this problem by exploring how existing pre-training methods impose relational structure in their induced per-sample latent spaces -- i.e., what constraints do pre-training methods impose on the distance or geometry between the pre-trained embeddings of two samples $\vec x_i$ and $\vec x_j$. Through a comprehensive review of existing pre-training methods, we find that this question remains open. This is true despite theoretical analyses demonstrating the importance of understanding this form of induced structure. Based on this review, we introduce a descriptive framework for pre-training that allows for a granular, comprehensive understanding of how relational structure can be induced. We present a theoretical analysis of this framework from first principles and establish a connection between the relational inductive bias of pre-training and fine-tuning performance. We also show how to use the framework to define new pre-training methods. We build upon these findings with empirical studies on benchmarks spanning 3 data modalities and ten fine-tuning tasks. These experiments validate our theoretical analyses, inform the design of novel pre-training methods, and establish consistent improvements over a compelling suite of baseline methods.
BibTeX
@article{mcdermott2023structure,
  author = {Matthew B. A. McDermott and Brendan Yap and Peter Szolovits and Marinka Žitnik},
  title = {{Structure Inducing Pre-Training}},
  journal = {Nat. Mac. Intell.},
  year = {2023},
  doi = {10.1038/s42256-023-00647-z},
}
Figure