OccamNet: A Fast Neural Model for Symbolic Regression at Scale

Owen Dugan, Rumen Dangovski, Allan dos Santos Costa, Samuel Kim, Pawan Goyal, Joseph M. Jacobson, Marin Soljačić

Abstract

Neural networks' expressiveness comes at the cost of complex, black-box models that often extrapolate poorly beyond the domain of the training dataset, conflicting with the goal of finding compact analytic expressions to describe scientific data. We introduce OccamNet, a neural network model that finds interpretable, compact, and sparse symbolic fits to data, à la Occam's razor. Our model defines a probability distribution over functions with efficient sampling and function evaluation. We train by sampling functions and biasing the probability mass toward better fitting solutions, backpropagating using cross-entropy matching in a reinforcement-learning loss. OccamNet can identify symbolic fits for a variety of problems, including analytic and non-analytic functions, implicit functions, and simple image classification, and can outperform state-of-the-art symbolic regression methods on real-world regression datasets. Our method requires a minimal memory footprint, fits complicated functions in minutes on a single CPU, and scales on a GPU.

Introduction

In the current era of artificial intelligence, researchers often rely on "black-box" neural networks. While these models excel at prediction, they frequently fail to provide insight into the underlying physical or mathematical laws governing a system. Symbolic Regression (SR) seeks to address this by identifying the specific analytic expressions—such as $F = G\frac{m_1m_2}{r^2}$—that describe a dataset.

Traditional SR methods, such as Genetic Programming, are often slow and difficult to scale, particularly when dealing with non-differentiable functions. Conversely, standard neural networks are highly expressive but lack the sparsity and interpretability required for scientific discovery. OccamNet bridges this gap by creating a model that is as interpretable as a mathematical formula yet as scalable and trainable as a neural network.

Methods

OccamNet treats symbolic regression as a probabilistic search through the space of possible functions. The architecture is built around several key innovations:

  • Symbolic Layers: Instead of standard linear layers, OccamNet utilizes symbolic layers composed of P-nodes (arguments) and image sublayers containing primitive functions like $\sin$, $\exp$, $+$, and $\times$.
  • Temperature-Controlled Connectivity: Each P-node probabilistically samples a single connection from the previous layer using a temperature-controlled softmax. This design enforces sparsity; a single forward pass represents one specific symbolic expression.
  • Sampling-Based Training: The model avoids backpropagating through the primitives themselves, allowing for the inclusion of non-differentiable functions like "IF" statements. Instead, it samples $R$ functions, evaluates their fitness, and uses a Risk-Seeking Policy Gradient loss to bias the probability distribution toward high-performing candidates.
OccamNet architecture showing (a) the full probabilistic connectivity and (b) a single sampled symbolic expression path.
  • Skip Connections: Similar to DenseNets, these allow the model to reuse sub-expressions from earlier layers, facilitating the construction of complex, nested formulas.
  • Constant Fitting: A two-step process is employed where the formula structure is identified via sampling, while numerical constants are optimized using standard gradient descent.

Evaluation and Results

The authors evaluated OccamNet across a diverse suite of benchmarks, including smooth analytic functions, non-analytic functions (e.g., sorting circuits, chaotic series), implicit functions (e.g., conservation of momentum), and image recognition tasks using MNIST and ImageNet.

Performance Benchmarks

OccamNet demonstrated significant advantages in both robustness and speed. It achieved non-zero success rates on every analytic benchmark tested, whereas competing methods like Eureqa and AI Feynman 2.0 failed entirely on at least one task. Furthermore, the model can fit complicated functions in minutes on a single CPU and offers up to an 8x speedup on GPUs for high-dimensional tasks.

Comparison of OccamNet against state-of-the-art symbolic regression methods across six analytic benchmarks, showing success rates and computation time.

Real-World and Vision Tasks

On the Penn Machine Learning Benchmarks (PMLB), OccamNet-GPU consistently outperformed Eplex, a state-of-the-art genetic algorithm, in terms of validation loss and training speed. When applied as a symbolic "head" for MNIST classification, the model discovered simple, interpretable rules based on specific pixel values, such as y = tanh(10(max(x_25,15, x_26,19) + ...)).

Discussion and Future Work

OccamNet introduces a powerful inductive bias for sparsity, adhering to "Occam’s Razor" by finding the simplest possible explanation for data. A key observation was "gradual modularity," where the model learns to identify and "lock in" useful sub-components before finding the full solution.

However, challenges remain. The search space for symbolic expressions grows exponentially with depth, making very deep expressions difficult to find. Additionally, the model requires a pre-defined set of primitive functions. Future research could explore meta-optimization to automatically select primitives and improved regularization techniques to prevent overfitting on small, noisy datasets.

Summary updated

Abstract

Neural networks' expressiveness comes at the cost of complex, black-box models that often extrapolate poorly beyond the domain of the training dataset, conflicting with the goal of finding compact analytic expressions to describe scientific data. We introduce OccamNet, a neural network model that finds interpretable, compact, and sparse symbolic fits to data, à la Occam's razor. Our model defines a probability distribution over functions with efficient sampling and function evaluation. We train by sampling functions and biasing the probability mass toward better fitting solutions, backpropagating using cross-entropy matching in a reinforcement-learning loss. OccamNet can identify symbolic fits for a variety of problems, including analytic and non-analytic functions, implicit functions, and simple image classification, and can outperform state-of-the-art symbolic regression methods on real-world regression datasets. Our method requires a minimal memory footprint, fits complicated functions in minutes on a single CPU, and scales on a GPU.
BibTeX
@misc{dugan2020occamnet,
  author = {Owen Dugan and Rumen Dangovski and Allan dos Santos Costa and Samuel Kim and Pawan Goyal and Joseph M. Jacobson and Marin Soljačić},
  title = {{OccamNet: A Fast Neural Model for Symbolic Regression at Scale}},
  year = {2020},
  eprint = {2007.10784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.10784},
}
Figure