Introduction
Symbolic Regression (SR) is the complex task of observing a set of data points $(x, y)$ and inferring the underlying mathematical law that generated them (e.g., $y = a \cdot \sin(bx) + c$). Unlike standard machine learning, which typically produces "black-box" models like deep neural networks, SR produces interpretable mathematical formulas essential for scientific discovery.
Researchers at Meta AI have developed a Transformer-based model that performs symbolic regression in a single "end-to-end" step. By predicting both the formula's structure and its numerical constants simultaneously, the model achieves performance competitive with state-of-the-art genetic programming while operating several orders of magnitude faster.
The Challenge of Symbolic Regression
Traditionally, SR is a grueling, computationally expensive two-step process:
- Skeleton Prediction: Finding the "shape" of the formula, including the operators and variables.
- Constant Optimization: Using non-convex optimization techniques (like BFGS) to find the best numerical constants for that skeleton.
The dominant approach to SR is Genetic Programming (GP), which evolves candidates by iterating this subroutine heavily. GP is notoriously slow and must start from scratch for every new problem. Previous deep learning attempts were limited to low-dimensional data and often failed because they only predicted the "skeleton." This left the difficult constant-fitting entirely to a solver that frequently got stuck in local minima.
An End-to-End Transformer Approach
To overcome these limitations, the authors propose an End-to-End (E2E) approach using a sequence-to-sequence Transformer.
Key innovations in this architecture include:
- Hybrid Vocabulary: Unlike previous models that only output math symbols, this model uses a hybrid vocabulary. It includes both operators (like
sin,exp,+) and numerical tokens (mantissa and exponent) to represent constants directly. - Permutation Invariance: The model uses a standard Encoder-Decoder Transformer but removes positional embeddings from the encoder. This ensures the model treats the input data points as an unordered set rather than a sequence.
- The Refinement Trick: While the model predicts constants directly, its initial guesses might lack perfect precision. The authors use the model's predicted constants as an "informed initialization" for a final BFGS optimization step. This "warm start" prevents the optimizer from getting lost in local minima.
- Scaling & Whitening: To handle data of varying scales, the input is "whitened" (normalized to mean 0 and variance 1). After prediction, the formula is mathematically "unscaled" to recover the original constants.
Key Insights and Model Behavior
The authors argue that predicting just the skeleton is an ill-posed problem that doesn't provide enough supervision during training. Predicting the full formula forces the model to learn a deeper relationship between the data's shape and its underlying logic.
Interestingly, visualizing the Transformer's attention heads reveals that the model performs sophisticated mathematical analysis. Some heads focus on local features like critical points, while others act similarly to Fourier transforms to detect periodicity. By using an "embedder" to compress high-dimensional input points into single vectors, the authors successfully scaled SR from the typical 3 input features seen in prior DL work up to 10 features.
Results: Speed and Accuracy
The model was trained on millions of randomly generated mathematical trees and evaluated on SRBench, a benchmark of 190 datasets including the Feynman Symbolic Regression Database and ODE-Strogatz problems.
The results highlight a massive leap in efficiency:
- Speed: The model provides a massive speedup, often three to four orders of magnitude faster than Genetic Programming. It generates solutions in seconds that might take a GP algorithm hours to find.
- Accuracy: On the SRBench Feynman problems, the model approaches the accuracy of state-of-the-art GP methods like Operon and significantly outperforms all previous deep-learning-based SR methods.
- Parsimony: The model tends to find simpler, more interpretable formulas than GP methods, which often produce overly complex, "bloated" expressions to achieve similar accuracy.
Limitations and Future Work
While this represents a major step forward, some hurdles remain. The model still doesn't quite match the absolute best GP algorithms on every single benchmark. Furthermore, scaling beyond 10 input dimensions remains a challenge for data generation and model training. Finally, the model is inherently biased by its synthetic training data; if a real-world physical law relies on a combination of operators the model hasn't seen, it may struggle.
Future work will likely focus on more sophisticated data generation protocols and adapting the model to handle the truly high-dimensional datasets common in modern scientific research.
Summary updated