Scaling Symbolic Regression for High-Dimensional Discovery

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2446 papers580 conceptsfrozen 2026-10-04

Symbolic Regression (SR) is often described as the "holy grail" of interpretable machine learning. Unlike neural networks, which function as high-dimensional black boxes, SR seeks to find the exact mathematical formula—like $F = G \frac{m_1 m_2}{r^2}$—that governs a dataset. As Peng Zeng and colleagues note, this provides a level of transparency and physical intuition that standard deep learning cannot match.

However, SR has historically been limited to "toy" problems with only a handful of variables. When moving from five features to five hundred, the search space of possible mathematical expressions doesn't just grow; it explodes. Recent research has focused on breaking through this "complexity wall" to enable automated scientific discovery in high-dimensional domains.

The Combinatorial Explosion

The fundamental challenge of SR is that mathematical expressions are discrete structures (trees). To find the right formula, an algorithm must decide which operators ($+, \sin, \exp$), constants, and variables to use, and in what order.

As Nan Jiang and Yexiang Xue explain, the number of possible combinations grows exponentially with the number of variables. If you have 100 input features, the likelihood of a random mutation in a Genetic Programming (GP) algorithm stumbling upon the correct functional form is statistically insignificant. This "curse of dimensionality" has traditionally relegated SR to simple, low-dimensional physics benchmarks.

Neural Networks as Symbolic Bridges

To scale SR, researchers are increasingly using deep learning not as the final model, but as a "scaffold" to simplify the problem. This hybrid approach leverages the ability of neural networks to handle high-dimensional data and then "distills" that knowledge into symbols.

Miles Cranmer and Peter W. Battaglia pioneered this by using Graph Neural Networks (GNNs). In their framework, a GNN is trained to model a complex system (like planetary orbits). Because GNNs have an inductive bias for relational data, they "factorize" the system into simpler internal components. SR is then applied only to these simplified internal functions, rather than the raw high-dimensional input. This allowed Pablo Lemos and Miles Cranmer to rediscover Newton’s Law of Gravitation from 30 years of noisy NASA trajectory data.

Similarly, Hongjue Zhao and Hairong Qi developed ScaleSR, which uses neural networks as surrogate data generators. By decomposing a multi-variable equation into a series of simpler single-variable problems, they reduced the search space enough to achieve a 78% recovery rate on complex benchmarks.

Recursive Decomposition and Symmetry

Another strategy for taming high dimensionality is to exploit the inherent structure of physical laws. Max Tegmark and Silviu-Marian Udrescu introduced AI Feynman, which uses neural networks to detect "hints" like symmetry, separability, and units.

If a neural network determines that a function is separable—meaning $f(x, y) = g(x) + h(y)$—the algorithm can split one high-dimensional problem into two smaller ones. This recursive "divide and conquer" strategy allows the system to solve equations from the Feynman Lectures on Physics that are far beyond the reach of brute-force search. This approach has since been extended to dynamical systems by Juliane Weilbach and colleagues to infer the structure of Ordinary Differential Equations (ODEs).

From Discrete Search to Continuous Optimization

A major bottleneck in traditional SR is that Genetic Programming is stochastic and non-differentiable. You cannot use gradient descent to "nudge" a sine wave into a cosine wave.

To solve this, Ahmed Alaa and Mihaela van der Schaar introduced symbolic metamodels. They use a highly general class of functions called Meijer G-functions, which can represent almost any standard mathematical operator. By parameterizing these functions, they transform the discrete search for a formula into a continuous optimization problem that can be solved via backpropagation.

Similarly, Maysum Panju and Ali Ghodsi developed the Multivariate Symbolic Function Learner (MSFL), which uses differentiable parse trees. This allows the model to "learn" the structure of a Partial Differential Equation (PDE) directly through gradient-based optimization, though scaling to extremely high-dimensional fluid dynamics remains an active area of research.

Efficient Search Architectures

Finally, new search paradigms are replacing the standard "horizontal" evolution of formulas.

  • Racing Schedules: Nan Jiang developed Racing-CVGP, which runs multiple experimental schedules in parallel and terminates unpromising ones early, significantly speeding up convergence in non-linear environments.
  • Vertical SR: Nan Jiang also proposed a "vertical" search strategy that bypasses exponential complexity by building equations through hierarchical layers rather than flat strings.
  • Transformer-based SR: Jonas Kulhanek introduced SymFormer, which treats SR as a sequence-to-sequence problem. By predicting symbols and numerical constants simultaneously, it achieves inference speeds orders of magnitude faster than evolutionary methods.

By combining the pattern-matching power of deep learning with the structural constraints of mathematics, these methods are transforming symbolic regression from a niche tool for small datasets into a robust engine for automated scientific discovery.

Papers in this version

Showing 60 of 446 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 352 papers
  2. Version 1 · 2026-10-04 · 446 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_combinatorial_search_space_explosion_sym_805284ca,
  title = {combinatorial search space explosion (symbolic search space complexity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/combinatorial-search-space-explosion-symbolic-search-space-complexity/rdr_805284ca10b8476d8906434edf31cddf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_symbolic_regression_for_high_dim_805284ca_v1,
  title = {Scaling Symbolic Regression for High-Dimensional Discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-symbolic-regression-for-high-dimensional-discovery/rdr_805284ca10b8476d8906434edf31cddf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}