Language Models Can Benefit from Domain-Specific Inductive Biases

Damien Teney, Liangze Jiang, Zachary Shinnick, Hemanth Saratchandran, Simon LuceyarXiv 2026 · arXiv

TL;DR

Modern Transformers are the "jacks-of-all-trades" of the AI world, but this paper argues they are masters of none. By replacing standard activation functions with domain-specific learned splines, the authors demonstrate that tailoring a model's inductive biases to specific data types—like code or mathematics—significantly improves training efficiency and generalization.

What Problem Does This Paper Address?

We currently train massive language models (LMs) using a "monolithic" approach: one architecture (the Transformer) and one set of hyperparameters to rule them all, regardless of whether the data is a Shakespearean play, a Python script, or a high school calculus problem.

While Transformers are remarkably versatile, the No-Free-Lunch theorem suggests that no single learning algorithm can be optimal for every task. The authors ask: How much performance are we leaving on the table by using a general-purpose architecture for such diverse data? Specifically, they investigate whether different subsets of pretraining data benefit from different inductive biases—the set of assumptions a model uses to predict outputs for previously unseen inputs.

Methods

The researchers focused on nonlinearities (activation functions like GeLU or Softmax) as the primary lever for adjusting inductive bias. Their approach involves a two-phase training pipeline:

  1. Warm-up Phase: Instead of using fixed functions, they use learned linear splines—flexible, piecewise-linear functions defined by a small number of parameters (typically 64). They train multiple models in a staggered schedule to optimize these splines on a "source" domain ($D_1$).
  2. Standard Training Phase: The learned splines are frozen and plugged into a new Transformer. This model is then trained from scratch on a "target" domain ($D_2$).
The experimental setup involves specializing a transformer for a source domain and then transferring those learned nonlinearities to a target domain.

This method allows them to "mix-and-match" architectures and datasets to see how well an inductive bias tuned for one domain (e.g., code) transfers to another (e.g., natural language).

Typical training curves during the warm-up phase, showing how nonlinearities evolve from constant functions to optimized splines.

Datasets

The study utilized seven representative subsets of standard pretraining mixtures:

  • Natural Language: FINEWEB (web text), ENWIK8 (Wikipedia), TINYSTORIES (children's stories), and SHAKESPEARE.
  • Code: CODESEARCHNET-JAVA and PYTHON.
  • Mathematics: DEEPMIND-MATH (56 types of high-school-level questions).

Key Insights and Ideas

  • Standard Transformers are Rarely Optimal: On every single dataset tested, the specialized variants outperformed the standard Transformer baselines in either training efficiency or final accuracy.
  • The "Math Gap": There is a clear divide in inductive biases. Architectures tuned for natural language transfer reasonably well to code, but math-tuned architectures perform poorly on natural language. This suggests that mathematical reasoning requires fundamentally different learning mechanisms than linguistic pattern matching.
  • Wider Loss Basins: The authors found that specialized architectures converge more smoothly. They suggest these models find "wider" minima in the loss landscape, making them less sensitive to hyperparameter choices like learning rate decay.
  • Abundant Lottery Tickets: Tuned nonlinearities seem to "prime" the model. Even at initialization with random weights, these models contain more useful circuits (or "winning lottery tickets") for the target task than standard models.

Key Results

The quantitative gains were consistent across the board. As seen in Figure 3, models optimized for their specific target dataset ($D_1 = D_2$) almost always sat at the top of the performance charts.

Validation performance across natural language, code, and math. Specialized architectures consistently outperform baselines.

In the mathematics domain, the benefits of specialization were even more pronounced. When training "specialist" models on individual math question types, the tuned nonlinearities improved accuracy by +3.5% to +7.5% compared to a +1.3% improvement for a "generalist" math model.

Comparison of generalist vs. specialist models on the MATH dataset, showing significant accuracy gains from learned nonlinearities.

Limitations and Future Work

The primary limitation is scale. The study was conducted on models ranging from 40M to 120M parameters. While the authors cite prior work suggesting these benefits persist at larger scales, it remains to be seen if the "monolithic" advantage of massive scale eventually washes out the benefits of domain-specific tuning. Additionally, the evaluation focused on perplexity and accuracy rather than downstream task performance.

The authors suggest that the future of LLMs might not be a single monolithic block, but rather a "Mixture of Inductive Biases." Instead of just Mixture-of-Experts (MoE) models where experts share the same architecture, we might see models where different experts utilize different architectures or optimizers tailored to the specific data they handle.

Summary updated

Abstract

Language models are typically trained with a single choice of architecture on diverse domains and data sources. Modern transformers have been extensively refined for such general-purpose learning, but are they equally effective across different subsets of the pretraining data? We investigate this question by training small language models from scratch on representative subsets of pretraining data such as web text, code, and mathematics. We introduce variants of transformers in which standard activation functions are replaced with learned splines tuned on specific datasets, providing a simple way to modify their inductive biases. Using a mix-and-match of architectures and datasets, we evaluate cross-domain transfer, i.e. how well does an architecture tuned for one domain perform on data from another? We find that the tuned transformer variants improve training efficiency or generalization on all seven of our tested datasets, but the gains are often domain-specific. For example, architectures tuned on a natural language dataset such as FineWeb perform well on TinyStories, whereas those tuned on mathematics perform poorly on natural language. We identify several mechanisms underlying these improvements, including wider loss basins and more abundant "lottery tickets". Our results show that different subsets of pretraining data can benefit from different learning mechanisms, highlighting a limitation of the prevailing trend towards a small set of general-purpose architectures and optimizers. This points to an opportunity to improve data efficiency by developing domain-specific learning strategies that retain the ability to train general-purpose models.
BibTeX
@misc{teney2026language,
  author = {Damien Teney and Liangze Jiang and Zachary Shinnick and Hemanth Saratchandran and Simon Lucey},
  title = {{Language Models Can Benefit from Domain-Specific Inductive Biases}},
  year = {2026},
  eprint = {2611.03756},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03756},
}
Figure