Fast Neural Models for Symbolic Regression at Scale

Allan dos Santos Costa, Rumen Dangovski, Owen Dugan, Samuel Kim, Pawan Goyal, Marin Soljačić, Joseph M. Jacobson

Abstract

Deep learning owes much of its success to the astonishing expressiveness of neural networks. However, this comes at the cost of complex, black-boxed models that extrapolate poorly beyond the domain of the training dataset, conflicting with goals of finding analytic expressions to describe science, engineering and real world data. Under the hypothesis that the hierarchical modularity of such laws can be captured by training a neural network, we introduce OccamNet, a neural network model that finds interpretable, compact, and sparse solutions for fitting data, a la Occam's razor. Our model defines a probability distribution over a non-differentiable function space. We introduce a two-step optimization method that samples functions and updates the weights with backpropagation based on cross-entropy matching in an evolutionary strategy: we train by biasing the probability mass toward better fitting solutions. OccamNet is able to fit a variety of symbolic laws including simple analytic functions, recursive programs, implicit functions, simple image classification, and can outperform noticeably state-of-the-art symbolic regression methods on real world regression datasets. Our method requires minimal memory footprint, does not require AI accelerators for efficient training, fits complicated functions in minutes of training on a single CPU, and demonstrates significant performance gains when scaled on a GPU.

Introduction

In the natural sciences, predictive accuracy is often secondary to understanding the underlying mechanisms. Standard deep learning models, while powerful, are essentially "black boxes" with millions of parameters. They excel at interpolation within a known domain but often fail to generalize or provide physical insight. Symbolic Regression (SR) addresses this by searching for the simplest mathematical expressions that explain a dataset. Traditional SR methods, such as Genetic Programming, frequently suffer from high computational costs and poor scalability. OccamNet is introduced to bridge the gap between the scalability of neural networks and the interpretability of symbolic expressions.

Methods

OccamNet treats a neural network as a generator of mathematical expressions. The architecture consists of "symbolic layers" containing P-nodes (placeholders) and an image sublayer with primitive functions like $\sin, \exp, +, \times$.

The OccamNet architecture samples symbolic expressions from a probability distribution over layers and updates weights based on the fitness of the resulting functions.

Instead of standard dense weights, OccamNet employs a $T$-softmax mechanism. During a forward pass, each P-node samples a single connection from the previous layer, creating a sparse Directed Acyclic Graph (DAG) that represents a specific formula. Because this sampling is discrete, the model uses Risk-Seeking Policy Gradients for training. It samples a batch of functions, evaluates their fitness, and uses backpropagation to increase the probability of paths that yield high-performing equations. Numerical constants are optimized via a two-step process: sampling the functional form first, then refining constants using gradient descent before updating architectural weights.

Results

The model was evaluated across analytic functions, programmatic tasks (e.g., sorting), implicit functions, and image recognition (MNIST/ImageNet). OccamNet demonstrated a built-in "Occam's Razor" bias, naturally favoring simpler expressions due to its sparse connectivity.

Comparison of OccamNet against baselines (Eureqa, Eplex, AIF, DSR) across six benchmark tasks, showing success rates and computation time.

OccamNet matched or exceeded the performance of state-of-the-art tools like Eureqa and AI Feynman. It was uniquely successful across the entire analytic function benchmark suite. Computationally, it fits complex functions in minutes on a single CPU and achieves an 8x speedup on high-dimensional tasks when scaled to a GPU. In symbolic vision tasks, it successfully replaced the classification head of a ResNet-50 with interpretable symbolic rules for ImageNet classes with minimal accuracy loss.

Conclusion

While OccamNet offers significant scalability and interpretability, challenges remain regarding the exponential growth of the search space for extremely deep equations and the dependency on a predefined library of primitives. Future work may involve meta-learning to automatically identify relevant primitives for specific domains. Nonetheless, OccamNet represents a significant step toward automated scientific discovery using scalable neural architectures.

Summary updated

Abstract

Deep learning owes much of its success to the astonishing expressiveness of neural networks. However, this comes at the cost of complex, black-boxed models that extrapolate poorly beyond the domain of the training dataset, conflicting with goals of finding analytic expressions to describe science, engineering and real world data. Under the hypothesis that the hierarchical modularity of such laws can be captured by training a neural network, we introduce OccamNet, a neural network model that finds interpretable, compact, and sparse solutions for fitting data, a la Occam's razor. Our model defines a probability distribution over a non-differentiable function space. We introduce a two-step optimization method that samples functions and updates the weights with backpropagation based on cross-entropy matching in an evolutionary strategy: we train by biasing the probability mass toward better fitting solutions. OccamNet is able to fit a variety of symbolic laws including simple analytic functions, recursive programs, implicit functions, simple image classification, and can outperform noticeably state-of-the-art symbolic regression methods on real world regression datasets. Our method requires minimal memory footprint, does not require AI accelerators for efficient training, fits complicated functions in minutes of training on a single CPU, and demonstrates significant performance gains when scaled on a GPU. Our implementation, demonstrations and instructions for reproducing the experiments are available at this https URL.
BibTeX
@misc{costa2020fast,
  author = {Allan dos Santos Costa and Rumen Dangovski and Owen Dugan and Samuel Kim and Pawan Goyal and Marin Soljačić and Joseph M. Jacobson},
  title = {{Fast Neural Models for Symbolic Regression at Scale}},
  year = {2020},
  eprint = {2007.10784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.10784},
}
Figure