Abstract
Deep learning models typically require vast amounts of data to achieve high performance. However, in domains such as bioinformatics and chemo-informatics, data is often scarce and expensive. This paper explores the use of inductive biases—mathematical prior knowledge integrated into model architectures—to improve performance in data-constrained settings. We discuss three primary contributions: Optimal Transport Kernel Embedding (OTKE) for sequences, GraphiT for graph-structured data, and Safe Sample Screening for efficient training. Experimental results demonstrate that these methods can outperform standard deep learning baselines and transfer learning approaches in small-data regimes.
Introduction
In the era of "Big Data," the prevailing wisdom suggests that more data is always better, as evidenced by the success of Large Language Models (LLMs) and massive vision models. However, many critical fields, including bioinformatics, rare disease research, and chemo-informatics, operate in "small data" environments where information is expensive, scarce, and highly structured.
The standard approach to data scarcity is transfer learning—pre-training on massive datasets and fine-tuning on target tasks. While effective, this approach is computationally intensive and often results in "black box" models that lack interpretability. This research investigates an alternative: building expressive models that learn effectively from scratch by incorporating inductive biases. These mathematical assumptions about data symmetry or topology narrow the search space during training, allowing models to achieve high performance with only a few hundred or thousand samples.
Methodological Contributions
Optimal Transport Kernel Embedding (OTKE)
For sequences and sets, such as proteins or text, we introduce a pooling layer based on Optimal Transport (OT). By utilizing Sinkhorn’s algorithm to align input features with a learned reference set, OTKE creates fixed-size representations compatible with various neural architectures. Unlike standard self-attention mechanisms which exhibit quadratic complexity ($O(n^2)$), OTKE scales linearly with sequence length, providing global context without prohibitive memory overhead.
GraphiT: Graph-Structured Transformers
Standard Transformers are permutation equivariant and often fail to capture the underlying topology of graph data. GraphiT addresses this by integrating graph kernels (e.g., diffusion or random walk kernels) to generate relative positional encodings. Furthermore, it utilizes Graph Convolutional Kernel Networks (GCKN) to enrich node features with local sub-structural information, such as paths and neighborhoods.
Safe Sample Screening
For convex models, we developed "SAFE rules" to identify and discard uninformative data points prior to training. By regularizing the dual optimization problem, the model identifies redundant samples that do not influence the decision boundary. This allows for significant dataset compression—up to 80% in some cases—without altering the final optimal solution.
Experimental Evaluation
The proposed methods were evaluated across several benchmarks:
- Bioinformatics: Protein fold recognition (SCOP 1.75/2.06) and chromatin profile prediction (DeepSEA).
- Graph Analytics: Molecular regression (ZINC) and graph classification (MUTAG, PROTEINS).
- Natural Language Processing: Sentiment analysis (SST-2).
Key Results
- Protein Folding: OTKE outperformed supervised baselines like CNNs and Set Transformers on the SCOP dataset. Notably, an unsupervised variant of OTKE surpassed many fully supervised models.
- Graph Regression: GraphiT achieved state-of-the-art results on the ZINC dataset, significantly reducing Mean Absolute Error compared to standard Graph Neural Networks (GNNs).
- Computational Efficiency: Screening rules successfully discarded up to 80% of samples on datasets like RCV-1 while maintaining exact model output, leading to substantial computational savings.
- Hybrid Integration: Combining OTKE with pre-trained models (e.g., ESM-1) improved accuracy, demonstrating that inductive biases can complement existing transfer learning paradigms.
Discussion and Future Work
The integration of kernels as a "prior" allows models to inherently understand data relationships—for instance, that nodes proximal in a graph should exert greater mutual influence. Moreover, the tie between attention scores and specific graph kernels in GraphiT enhances interpretability, allowing researchers to visualize which chemical groups contribute to specific molecular properties.
Future research will focus on scaling GraphiT to larger graphs, as the self-attention mechanism remains a bottleneck. Additionally, extending "Safe Screening" guarantees to non-convex deep learning architectures remains a significant challenge. Ultimately, these data-efficient, interpretable tools are poised to accelerate scientific discovery in lab-based AI environments.
Summary updated