Abstract
Symbolic Regression (SR) is a powerful technique for automatically discovering mathematical expressions from input data. Mainstream SR algorithms search for the optimal symbolic tree in a vast function space, but the increasing complexity of the tree structure limits their performance. Inspired by neural networks, symbolic networks have emerged as a promising new paradigm. However, most existing symbolic networks still face certain challenges: binary nonlinear operators {×, ÷} cannot be naturally extended to multivariate operators, and training with fixed architecture often leads to higher complexity and overfitting. In this work, we propose a Unified Symbolic Network that unifies nonlinear binary operators into nested unary operators and define the conditions under which UniSymNet can reduce complexity. Moreover, we pre-train a Transformer model with a novel label encoding method to guide structural selection, and adopt objective-specific optimization strategies to learn the parameters of the symbolic network. UniSymNet shows high fitting accuracy, excellent symbolic solution rate, and relatively low expression complexity, achieving competitive performance on low-dimensional Standard Benchmarks and high-dimensional SRBench.
Introduction
Symbolic Regression (SR) aims to find a mathematical formula that describes a dataset, effectively discovering the underlying physical laws. While traditional methods like Genetic Programming (GP) are widely used, they often struggle with high-dimensional problems and the production of overly complex, "bloated" expressions. Recently, symbolic networks (e.g., Equation Learner or EQL) have attempted to bridge the gap between neural networks and symbolic math by using mathematical operators as activation functions.
However, these models face two major hurdles. First, binary operators like multiplication (×) and division (÷) are difficult to extend to multivariate inputs within standard neural layers. Second, training with a fixed architecture often leads to overfitting or unnecessary complexity. UniSymNet addresses these issues by introducing a unified representation that simplifies the network's vocabulary and a Transformer-based guide that selects the optimal structure before parameter optimization.
The UniSymNet Framework
Unary Unification via the Ψ Representation
The core innovation of UniSymNet is the $\Psi$ representation. This mathematical approach leverages properties such as $\ln(x \cdot y) = \ln x + \ln y$ to transform binary operators (×, ÷, pow) into nested unary operators (ln, exp). This allows the network to handle "multi-ary" operations—multiplying multiple variables simultaneously—through simple linear combinations in hidden layers.
Theoretical proofs provided by the authors demonstrate that this representation allows UniSymNet to represent common physical laws using fewer nodes and shallower depths than traditional EQL-type networks, enhancing both efficiency and interpretability.
Transformer-Guided Structure Learning
UniSymNet employs a bi-level optimization strategy. In the outer level, an encoder-decoder Transformer is pre-trained on millions of synthetic equations. At test time, this model takes a small set of data points and predicts a "sparse encoding," which serves as a blueprint for the most likely network structure. This approach allows the model to leverage prior knowledge from a massive synthetic dataset (1.5M to 3.75M functions), significantly accelerating the search process compared to methods that start from scratch.
Optimization Strategies
Once the structure is determined, UniSymNet uses objective-specific strategies to optimize parameters:
- Symbolic Function Optimization (SFO): This strategy focuses on finding discrete exponents and is particularly robust to noisy data, making it ideal for scientific discovery.
- Differentiable Network Optimization (DNO): This involves training the network like a standard deep learning model using gradient descent and pruning techniques.
Key Results and Discussion
UniSymNet was evaluated on standard low-dimensional benchmarks and the rigorous high-dimensional SRBench, which includes equations from the Feynman Symbolic Regression Database.
- State-of-the-Art Performance: On the Feynman datasets, UniSymNet achieved the highest symbolic solution rate among 14 competing "out-of-domain" methods, including AIFeynman and DSR.
- Interpretability and Complexity: The model consistently produced expressions with lower complexity (fewer operators and constants) while maintaining high fitting accuracy ($R^2 > 0.99$).
- Robustness and Extrapolation: The SFO strategy proved highly robust to noise. Furthermore, because UniSymNet discovers the underlying functional form, it generalizes far better to data ranges outside the training set than black-box models like MLPs.
Conclusion
UniSymNet represents a significant step forward in symbolic regression by unifying complex operators and utilizing a Transformer to guide structural search. By combining the strengths of neural networks with the interpretability of symbolic math, it offers a powerful tool for scientific discovery across various domains. Future work includes extending the framework to Partial Differential Equations (PDEs) and improving numerical stability for a wider range of operators.
Summary updated