Abstract
While language model pre-training has fundamentally shifted the machine learning landscape, the mechanisms by which it improves downstream performance—particularly for whole-sample embeddings—remain poorly understood. This is especially evident when adapting these methods to non-linguistic domains like biology or medicine. We introduce Structure Inducing Pre-Training (SIPT), a framework that explicitly imposes relational structure on latent spaces using external graphs. By decomposing pre-training objectives into intra-sample and structure-inducing components, we provide a theoretical foundation for how relational inductive bias influences fine-tuning performance. Our empirical results across three data modalities and ten tasks demonstrate that SIPT consistently outperforms standard pre-training baselines, offering a roadmap for more intentional transfer learning design.
Introduction
In Natural Language Processing (NLP), pre-training typically focuses on "per-token" objectives, such as predicting a masked word. When these models are applied to "per-sample" tasks—like classifying the sentiment of an entire document—they rely on the model's internal ability to aggregate token-level features into a coherent global embedding. However, in domains such as proteomics or clinical informatics, the primary unit of interest is often the whole sample (e.g., a protein sequence or a patient record).
Current pre-training methods often fail to impose explicit or deep structural constraints on these per-sample latent spaces. While contrastive learning offers some implicit geometric constraints through data augmentation, it lacks a mechanism to reason about the precise distances between disparate samples. SIPT addresses this gap by leveraging domain-specific knowledge, represented as graphs, to shape the geometry of the embedding space.
The SIPT Framework
The core of the SIPT framework is the decomposition of the pre-training loss into two distinct objectives:
- Intra-sample Objective ($\mathcal{L}_M$): This represents standard self-supervised tasks, such as Masked Language Modeling (MLM), which focus on the internal structure of a single data point.
- Structure-inducing Objective ($\mathcal{L}_{SI}$): This new term regularizes the latent space geometry based on a user-defined Pre-Training Graph ($G_{PT}$).
By utilizing $G_{PT}$, researchers can encode known relationships directly into the model's training signal. For instance, if two proteins are known to interact in a biological pathway, an edge in $G_{PT}$ ensures the $\mathcal{L}_{SI}$ term pushes their embeddings closer together. We implement this using Structure-Preserving Metric Learning (SPML), adapting multi-similarity and contrastive losses to ensure the latent space reflects the connectivity and topology of the input graph.
Theoretical Foundations: Explicit and Deep Constraints
We propose a taxonomy to categorize pre-training constraints based on two dimensions:
- Explicitness: A constraint is explicit if it allows for direct reasoning about the distance or similarity between any two samples in the dataset.
- Depth: A constraint is deep if it requires a high-dimensional manifold to be satisfied, preventing the model from simply collapsing data into a few discrete clusters.
Our theoretical analysis culminates in Theorem 1 (The Performance Floor), which establishes a mathematical link between the pre-training graph and downstream success. We prove that if the $G_{PT}$ aligns with the downstream task (a property termed "Local Consistency"), the nearest-neighbor accuracy of the fine-tuned model is lower-bounded by that consistency. Essentially, a well-constructed pre-training graph provides a guaranteed performance baseline.
Experimental Results
We validated SIPT across three distinct modalities and ten fine-tuning tasks, comparing it against standard per-token and per-sample baselines:
- Proteomics: Using the "Tree-of-life" dataset, we utilized a protein-protein interaction network as $G_{PT}$. SIPT achieved a 13.1% relative reduction in error (RRE) on protein stability tasks.
- Biomedical Text: Using citation networks from the Microsoft Academic Graph to guide the pre-training of abstract embeddings, SIPT reached an RRE of 17.7% on specific classification tasks.
- Network Analysis: Using protein-protein interaction ego-graphs, we demonstrated that SIPT creates more meaningful clusters in the latent space compared to standard masked pre-training.
Beyond raw accuracy, SIPT models exhibited faster convergence, reaching peak performance levels significantly earlier in the fine-tuning process than baseline models. Visualizations via UMAP confirmed that SIPT induces a latent space with clearer, more biologically or logically relevant clusters.
Conclusion
SIPT demonstrates that the "black box" of pre-training can be opened and guided by explicit relational structures. While the success of the method depends on the quality and relevance of the pre-training graph, it provides a powerful tool for domains where relational data is abundant but labeled task data is scarce. Future work will focus on automating graph discovery and exploring higher-order topological features to further refine the geometry of induced latent spaces.
Summary updated