Structure-inducing pre-training

Matthew B. A. McDermott, Brendan Yap, Peter Szolovits, Marinka Zitnik

Introduction

The pre-training and fine-tuning (PT/FT) paradigm has become the cornerstone of modern machine learning, particularly in natural language processing. However, applying these methods to specialized domains like biology or medicine often reveals a lack of understanding regarding why certain pre-training tasks translate to downstream success. Researchers have identified that standard pre-training often fails to impose explicit structural constraints on the model's latent space, leading to suboptimal representations.

The Problem: Implicit and Shallow Constraints

Current pre-training methods typically rely on implicit constraints (like standard contrastive learning) or shallow objectives (like simple classification). These approaches often result in an "anisotropic" latent space where data is unevenly distributed, making it difficult for models to generalize to new, complex tasks. To address this, there is a need for "deep and explicit" structural constraints that guide the model's internal map of the data.

The SIPT Framework

The proposed Structure-Inducing Pre-training (SIPT) framework explicitly shapes the geometry of a model's latent space using a pre-training graph ($G_{PT}$). The framework optimizes a dual-objective loss function:

  1. Masking Objective ($\mathcal{L}_M$): A standard self-supervised task, such as masked language modeling, that forces the model to learn internal features of individual samples.
  2. Structure-Inducing Objective ($\mathcal{L}_{SI}$): A relational constraint that uses the pre-training graph to define proximity. If two samples are linked in the graph (e.g., interacting proteins or cited papers), the model is incentivized to place their embeddings closer together.
Overview of the SIPT framework, combining masking and structure-inducing objectives.

By combining these objectives, SIPT ensures the model learns both the "what" (features) and the "where" (relational context) of the data. Crucially, the authors provide mathematical proof that if the pre-training graph reflects the underlying data manifold, downstream performance is theoretically guaranteed to improve as the dataset grows.

Experimental Evaluation

SIPT was evaluated across three diverse domains to test its versatility:

  • Proteins: Utilizing protein-protein interaction graphs from the "Tree of Life" dataset (~1.5M sequences).
  • Scientific Abstracts: Using citation networks from the Microsoft Academic Graph (~650k abstracts).
  • Biological Networks: Leveraging Gene Ontology-based similarity in protein-protein interaction ego-graphs.

Key Results and Convergence

SIPT matched or outperformed competitive baselines in 10 out of 15 downstream tasks. Notably, it achieved a 17.7% error reduction in citation intent tasks and a 7-8% reduction in protein remote homology prediction.

Beyond final accuracy, SIPT demonstrates superior convergence properties. As shown in the training curves, SIPT models reach higher performance levels significantly faster than masking-only or multi-task baselines. This suggests that the structural inductive bias provided by the graph allows the model to find optimal solutions more efficiently.

Comparison of fine-tuning performance (AUROC) across different pre-training strategies over training iterations.

Conclusion

Structure-Inducing Pre-training represents a shift from task-centric to geometry-centric pre-training. By providing explicit inductive biases through relational graphs, SIPT creates more robust and structured latent spaces. While the framework depends on high-quality graph data, its ability to bridge domain knowledge and deep learning offers a powerful new tool for non-linguistic data modalities. Future work will explore the use of higher-order structures like hypergraphs to further refine these mathematical maps.

Summary updated

Abstract

Language model pre-training and the derived general-purpose methods have reshaped machine learning research. However, there remains considerable uncertainty regarding why pre-training improves the performance of downstream tasks. This challenge is pronounced when using language model pre-training in domains outside of natural language. Here we investigate this problem by analysing how pre-training methods impose relational structure in induced per-sample latent spaces—that is, what constraints do pre-training methods impose on the distance or geometry between the pre-trained embeddings of samples. A comprehensive review of pre-training methods reveals that this question remains open, despite theoretical analyses showing the importance of understanding this form of induced structure. Based on this review, we introduce a pre-training framework that enables a granular and comprehensive understanding of how relational structure can be induced. We present a theoretical analysis of the framework from the first principles and establish a connection between the relational inductive bias of pre-training and fine-tuning performance. Empirical studies spanning three data modalities and ten fine-tuning tasks confirm theoretical analyses, inform the design of novel pre-training methods and establish consistent improvements over a compelling suite of methods. Designing methods to induce explicit and deep structural constraints in latent space at the sample level is an open problem in natural language processing-derived methods relying on transfer learning. McDermott and colleagues propose and analyse a pre-training framework imposing such structural constraints, and empirically demonstrate its advantages by showing that it outperforms existing pre-training state-of-the-art methods.
BibTeX
@article{mcdermott2023structure,
  author = {Matthew B. A. McDermott and Brendan Yap and Peter Szolovits and Marinka Zitnik},
  title = {{Structure-inducing pre-training}},
  journal = {Nat. Mac. Intell.},
  year = {2023},
  doi = {10.1038/s42256-023-00647-z},
}
Figure