Interpretable Neuroevolutionary Models for Learning Non-Differentiable Functions and Programs

Allan Costa, Rumen Dangovski, Samuel Kim, Pawan Goyal, Marin Soljacic, Joseph Jacobson

Abstract

Deep learning owes much of its success to the astonishing expressiveness of neural networks. However, this comes at the cost of complex, black-boxed models that extrapolate poorly beyond the domain of the training dataset, conflicting with goals of finding analytic expressions to describe science, engineering and real world data. Under the hypothesis that the hierarchical modularity of such laws can be captured by training a neural network, we introduce OccamNet, a neural network model that finds interpretable, compact, and sparse solutions for fitting data, à la Occam's razor. Our model defines a probability distribution over a non-differentiable function space. We introduce a two-step optimization method that samples functions and updates the weights with backpropagation based on cross-entropy matching in an evolutionary strategy: we train by biasing the probability mass toward better fitting solutions. OccamNet is able to fit a variety of symbolic laws including simple analytic functions, recursive programs, implicit functions, simple image classification, and can outperform noticeably state-of-the-art symbolic regression methods on real world regression datasets. Our method requires minimal memory footprint, does not require AI accelerators for efficient training, fits complicated functions in minutes of training on a single CPU, and demonstrates significant performance gains when scaled on a GPU. Our implementation, demonstrations and instructions for reproducing the experiments are available at https://github.com/druidowm/OccamNet_Public.

Introduction

Standard deep learning models are often "black boxes"—highly expressive but offering little insight into the underlying physical or mathematical laws governing a dataset. They frequently fail to generalize outside their training domain. Symbolic Regression (SR) aims to address this by discovering the simplest mathematical expression (e.g., $y = \sin(x) + e^x$) that describes the data. However, traditional SR methods, such as Genetic Programming, are often slow and difficult to scale to high-dimensional data. Previous neural-based SR attempts often required differentiable functions for backpropagation, excluding many useful discrete mathematical operations like "if-else" logic or sorting circuits.

Methodology

OccamNet bridges the gap between connectionist neural networks and symbolic search by treating the network as a trainable probability distribution over mathematical expressions. The architecture consists of symbolic layers containing a library of primitive functions (e.g., $+$, $\times$, $\sin$, $\exp$, $\text{max}$).

Instead of dense linear weights, OccamNet utilizes temperature-controlled connectivity via a $T$-softmax layer. Each node probabilistically samples exactly one input edge from the previous layer, forcing the network to represent a Sparse Directed Acyclic Graph (DAG) that corresponds directly to a mathematical formula. This architecture naturally prefers simpler, more compact expressions, adhering to the principle of Occam's Razor.

OccamNet architecture showing (a) the full connectivity and (b) a sampled sparse symbolic path.

Training is performed using Risk-Seeking Policy Gradients. Rather than backpropagating through the primitive functions themselves, the model samples thousands of potential functions, evaluates their fitness against the data, and uses a cross-entropy loss to bias the probability mass toward the best-performing paths. This allows the inclusion of non-differentiable or implicit functions. For expressions requiring unknown constants, the model uses a two-step process: it first samples the functional form, then performs gradient descent to optimize constants for that specific form.

Evaluation and Results

The authors evaluated OccamNet across a diverse suite of benchmarks, including analytic functions, recursive programs, implicit functions (such as conservation of momentum), and real-world regression datasets from the Penn Machine Learning Benchmarks (PMLB).

Performance comparison of OccamNet against state-of-the-art symbolic regression methods across six analytic functions.

Key results include:

  • Robustness: OccamNet was the only method tested that achieved a non-zero success rate on every analytic function in the benchmark, outperforming Eureqa, AI Feynman, and DSR.
  • Speed: On real-world datasets, GPU-accelerated OccamNet was often more than an order of magnitude faster than evolutionary baselines like Eplex.
  • Interpretability in Vision: When used as a "head" for a pretrained ResNet, it discovered simple symbolic rules for image classification while maintaining competitive accuracy.
  • Implicit Discovery: It successfully identified physical laws from raw data, a task where traditional symbolic regression tools struggled without specific data ordering.

Discussion and Limitations

While OccamNet is efficient and highly parallelizable on GPUs, the search space grows exponentially with network depth, making extremely nested equations a challenge. Additionally, the model requires a predefined set of primitive functions; if a necessary operation is missing from the library, an exact fit cannot be found. Future work may involve meta-learning to automatically identify domain-specific primitives and further optimizing the sampling process to ignore unused connections.

Summary updated

Abstract

Deep learning owes much of its success to the astonishing expressiveness of neural networks. However, this comes at the cost of complex, black-boxed models that extrapolate poorly beyond the domain of the training dataset, conflicting with goals of finding analytic expressions to describe science, engineering and real world data. Under the hypothesis that the hierarchical modularity of such laws can be captured by training a neural network, we introduce OccamNet, a neural network model that finds interpretable, compact, and sparse solutions for fitting data, \`{a} la Occam's razor. Our model defines a probability distribution over a non-differentiable function space. We introduce a two-step optimization method that samples functions and updates the weights with backpropagation based on cross-entropy matching in an evolutionary strategy: we train by biasing the probability mass toward better fitting solutions. OccamNet is able to fit a variety of symbolic laws including simple analytic functions, recursive programs, implicit functions, simple image classification, and can outperform noticeably state-of-the-art symbolic regression methods on real world regression datasets. Our method requires minimal memory footprint, does not require AI accelerators for efficient training, fits complicated functions in minutes of training on a single CPU, and demonstrates significant performance gains when scaled on a GPU. Our implementation, demonstrations and instructions for reproducing the experiments are available at https://github.com/druidowm/OccamNet_Public.
BibTeX
@misc{costa2020interpretable,
  author = {Allan Costa and Rumen Dangovski and Samuel Kim and Pawan Goyal and Marin Soljacic and Joseph Jacobson},
  title = {{Interpretable Neuroevolutionary Models for Learning Non-Differentiable Functions and Programs}},
  year = {2020},
  eprint = {2007.10784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.10784},
}
Figure