Compositional Kernel Search for Automated Statistical Modeling

This direction is split into other directions: compositional kernel grammar. Its last version stays readable and citable here.

Version 1 of 143 papers50 conceptsfrozen 2026-10-04

Gaussian Processes (GPs) are a cornerstone of Bayesian machine learning, prized for their ability to provide principled uncertainty estimates. However, their performance depends entirely on the choice of a covariance kernel, which defines the prior assumptions about the data's shape. Traditionally, selecting this kernel was a "black art" requiring expert intuition to decide whether a dataset exhibited linear trends, periodicity, or smoothness.

Researchers have transformed this manual engineering task into an automated search problem. By defining a formal grammar of kernels, systems can now discover the underlying structure of complex datasets—ranging from global economic trends to quantum phase transitions—without human intervention.

A Grammar for Model Structures

The core intuition behind this research direction, pioneered by David Duvenaud, James Robert Lloyd, and Zoubin Ghahramani in Structure Discovery in Nonparametric Regression through Compositional Kernel Search, is that complex patterns are simply compositions of simpler ones.

Just as a language uses a grammar to combine words into sentences, this framework uses a compositional grammar to combine base kernels (like Linear, Periodic, or Squared Exponential) using algebraic operators:

  • Addition ($+$): Represents the sum of independent variations. If a dataset has a long-term linear trend and seasonal fluctuations, adding a Linear kernel to a Periodic kernel captures both. As noted in Beyond Intuition, a Framework for Applying GPs to Real-World Data, addition acts like a logical "OR."
  • Multiplication ($\times$): Represents interactions or modulations. Multiplying a Periodic kernel by a Squared Exponential kernel creates a "locally periodic" model—one where the rhythm stays the same but the amplitude or shape changes over time. This acts like a logical "AND," capturing the intersection of features.

By treating kernels as symbolic expressions, the system can represent intricate structures like "a linear trend with a yearly cycle that gradually decays in intensity."

Searching the Model Space

With a grammar in place, the problem shifts from "What kernel should I use?" to "How do I find the best expression in this language?"

In Automatic Construction and Natural-Language Description of Nonparametric Regression Models, the authors implement a greedy search. The algorithm starts with a simple base kernel and iteratively proposes expansions (e.g., replacing a kernel $K$ with $K + \text{Base}$ or $K \times \text{Base}$).

To prevent the model from becoming over-complex and "overfitting" the noise, the search is guided by the Bayesian Information Criterion (BIC). The BIC rewards models that fit the data well but penalizes those with too many parameters, ensuring the discovered structure remains parsimonious and interpretable. This approach allows the "Automatic Statistician" to not only fit the data but also generate natural-language reports explaining the discovered components (e.g., "This component represents a medium-term cycle").

Physical Extrapolation and Quantum Discovery

The power of compositional search extends beyond simple curve fitting into the realm of scientific discovery. John Sous and Roman V. Krems demonstrated in Extrapolating Quantum Observables with Machine Learning that these models can perform "physical extrapolation."

By discovering the correct functional form of a quantum system's observables within a single phase, the GP can predict sharp quantum phase transitions that lie entirely outside the training data. Because the compositional search finds the "right" mathematical structure rather than just interpolating points, it can infer the existence of transitions that are hidden behind other intermediate phases.

From Symbolic Search to Probabilistic Programs

Recent work has moved toward even more flexible representations. Feras Saad and Vikash Mansinghka frame structure discovery as probabilistic program synthesis. In Time Series Structure Discovery via Probabilistic Program Synthesis, they represent model structures as abstract syntax trees.

This allows for more sophisticated search techniques, such as Sequential Monte Carlo (SMC). Instead of a single greedy path, AutoGP maintains a population of potential model structures, allowing it to explore multiple hypotheses simultaneously. This probabilistic approach provides better-calibrated uncertainty; if the data is ambiguous, the model can remain uncertain about whether a trend is truly periodic or just a series of random shifts.

By bridging symbolic logic (the grammar) with statistical inference (the Gaussian Process), this research direction enables machines to move from "black box" prediction to "white box" understanding of the patterns that govern the world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_kernel_search_for_automate_92911938,
  title = {Compositional Kernel Search for Automated Statistical Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-kernel-search-for-automated-statistical-modeling/rdr_92911938bec648719326a25b954f53d7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_kernel_search_for_automate_92911938_v1,
  title = {Compositional Kernel Search for Automated Statistical Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-kernel-search-for-automated-statistical-modeling/rdr_92911938bec648719326a25b954f53d7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}