End-to-end symbolic regression with transformers

Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample, François Charton

Abstract: Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the "skeleton" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this paper, we challenge this two-step procedure, and task a Transformer to directly predict the full mathematical expression, constants included. One can subsequently refine the predicted constants by feeding them to the non-convex optimizer as an informed initialization. We present ablations to show that this end-to-end approach yields better results, sometimes even without the refinement step. We evaluate our model on problems from the SRBench benchmark and show that our model approaches the performance of state-of-the-art genetic programming with several orders of magnitude faster inference.

Introduction

Symbolic Regression (SR) aims to discover human-readable mathematical laws, such as $f(x) = a \cdot \sin(bx) + c$, from empirical data. Unlike standard "black-box" machine learning models, SR provides interpretable expressions that are crucial for scientific discovery. Traditionally, this problem is solved using Genetic Programming (GP), which evolves formulas over thousands of iterations. While accurate, GP is computationally expensive and lacks the ability to learn from previous tasks.

Recent deep learning approaches have attempted to bridge this gap using a two-step process: first predicting a "skeleton" (the formula structure without constants) and then employing a classical optimizer (like BFGS) to determine the numerical values. However, this approach is often ill-posed, as different skeletons can appear identical in data, and optimizers frequently converge to local minima without a precise starting guess.

Methodology

This research introduces an End-to-End (E2E) Transformer architecture designed to predict both the symbolic structure and the numerical constants simultaneously.

The proposed training and inference pipelines for end-to-end symbolic regression.

Architecture and Training

The model utilizes a sequence-to-sequence Transformer with an asymmetric design: a compact 4-layer encoder and a deeper 16-layer decoder. To process high-dimensional inputs (up to 10 variables), a 2-layer Feed-Forward Network (FFN) embedder maps each input point $(x, y)$ to a single embedding. The model employs a hybrid vocabulary consisting of symbolic operators (e.g., +, exp, sin) and numeric tokens representing constants in base-10 floating-point notation.

The model is pre-trained on a massive synthetic dataset of millions of equations. During training, the inclusion of constants provides a richer supervisory signal, which counter-intuitively helps the model learn the underlying mathematical structure more effectively than training on skeletons alone.

Inference and Refinement

During inference, the model generates candidate formulas. While the Transformer predicts constants directly, these are treated as an "informed guess." A single subsequent pass of the BFGS optimizer is used to refine these constants, significantly boosting precision. The system also employs "bagging"—splitting large datasets into smaller chunks—to generate multiple candidates and select the best fit.

Experimental Results

The model was evaluated using the SRBench benchmark, encompassing Feynman physics problems, black-box regression datasets from PMLB, and ODE-Strogatz trajectories.

Comparison of inference time vs. accuracy across different benchmarks. The proposed model ("Ours") achieves competitive accuracy with significantly lower inference times.

Performance and Efficiency

As shown in the results above, the E2E Transformer approaches the accuracy of state-of-the-art GP methods like Operon or SBP-GP on the Feynman benchmarks while being several orders of magnitude faster. Inference typically completes in seconds, compared to the minutes or hours required by traditional GP. Furthermore, the model produces formulas with lower complexity (fewer nodes) than top GP methods, enhancing interpretability.

Scaling and Analysis

This model successfully scales to 10 input dimensions, a significant improvement over previous deep learning SR models which were often limited to 3. Analysis of the encoder's attention maps reveals that the model naturally learns to perform mathematical analysis, with specific heads focusing on periodicity, singularities, or extrema.

Discussion and Limitations

While the model represents a significant step forward, challenges remain. The current architecture is sensitive to noise in the labels when the refinement step is omitted. Additionally, while 10 dimensions is a milestone for neural SR, many real-world scientific problems involve hundreds of variables. The model also showed lower performance on the Strogatz datasets, likely because these time-ordered trajectories differ from the independent and identically distributed (i.i.d.) points used during training. Future work will focus on improving zero-shot robustness to noise and adapting the architecture for truly high-dimensional, non-i.i.d. data.

Summary updated

Abstract

Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the "skeleton" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this paper, we challenge this two-step procedure, and task a Transformer to directly predict the full mathematical expression, constants included. One can subsequently refine the predicted constants by feeding them to the non-convex optimizer as an informed initialization. We present ablations to show that this end-to-end approach yields better results, sometimes even without the refinement step. We evaluate our model on problems from the SRBench benchmark and show that our model approaches the performance of state-of-the-art genetic programming with several orders of magnitude faster inference.
BibTeX
@inproceedings{kamienny2022end,
  author = {Pierre-Alexandre Kamienny and Stéphane d'Ascoli and Guillaume Lample and François Charton},
  title = {{End-to-end symbolic regression with transformers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=O7m250pGGw},
}
Figure