NOMTO: Neural Operator-based symbolic Model approximaTion and discOvery

Sergei Garmaev, Siddhartha Mishra, Olga FinkarXiv 2025 · DOI · arXiv

Introduction

Discovering the mathematical equations that govern physical systems is a fundamental goal of scientific computing. This field, known as Symbolic Regression (SR), aims to transform raw observational data into concise, interpretable formulas. However, traditional SR methods often struggle with complex operations such as derivatives, singularities (e.g., $1/x$), or special functions (e.g., the Gamma function).

Researchers from EPFL and ETH Zurich have introduced NOMTO (Neural Operator-based symbolic Model approximaTion and discOvery), a framework that leverages Neural Operators to overcome these limitations. NOMTO treats complex mathematical operations as learnable surrogates, enabling the discovery of governing physical laws—such as the Heat and Burgers' equations—directly from data.

The Challenge of Symbolic Regression

Most physical processes are best described by non-linear symbolic models. While deep learning excels at prediction, it often produces "black box" models that lack physical interpretability. Symbolic Regression seeks to bridge this gap by searching for explicit equations.

Current SR approaches typically fall into two categories:

  1. Search-based (e.g., Genetic Programming): These methods are flexible but often rely on evolutionary heuristics that struggle with high-dimensional data and complex operators.
  2. Model-based (e.g., EQL or KANs): These use gradient descent to find equations but are generally restricted to simple, continuous algebraic functions. They often fail when formulas involve derivatives or singularities.

NOMTO addresses this by providing a method that is as efficient as gradient descent but as flexible as a physicist’s toolkit.

Methodology: The NOMTO Framework

The core innovation of NOMTO is the use of Neural Operators (specifically Fourier Neural Operators and Convolutional Neural Operators) as surrogate library operations. The framework operates in four distinct steps:

1. Pre-training Surrogates

Instead of using raw functions, NOMTO trains a Neural Operator (NO) to approximate specific operations. Because NOs operate on functional spaces (discretized grids), they can approximate derivatives and integrals more effectively than standard point-wise networks.

2. Building the Graph

The trained NO blocks are arranged into a computational graph. Each edge in the graph has a weight that acts as a multiplier, determining the contribution of each operator to the final model.

Steps 1 and 3 of the NOMTO workflow: training NO surrogates and stacking them into a computational graph.

3. Sparse Optimization

The system is trained to minimize the difference between the graph's output and the target data. To ensure the resulting formula is interpretable, a modified $\ell_{1/2}$ penalty is used alongside periodic pruning to eliminate unnecessary connections.

4. Symbolic Extraction

Once the weights are optimized, the dominant path through the graph is converted back into a standard mathematical expression using the SymPy library.

Steps 2 and 4 of the NOMTO workflow: composing the sub-graph and extracting the final symbolic expression.

Key Insights and Results

NOMTO introduces several critical concepts for robust symbolic discovery:

  • Differentiable Physics: By using neural surrogates for derivatives, NOMTO allows the optimizer to backpropagate through calculus, turning the search for differential equations into a gradient-based optimization problem.
  • Taming Singularities: Functions like $\ln(x)$ or $1/x$ are redefined to be well-defined across the entire support during training, preventing numerical instabilities.
  • Operator Selection: The researchers found that Fourier Neural Operators (FNO) excel at handling discontinuities, while Convolutional Neural Operators (CNO) are more accurate for smooth physical systems.

Performance Highlights

  • Benchmark Success: NOMTO performed comparably to state-of-the-art methods like KAN and EQL on standard algebraic benchmarks (Nguyen and Keijzer sets).
  • Calculus Discovery: NOMTO successfully identified expressions containing 1D derivatives and complex special functions (e.g., Airy and Gamma functions) with $R^2$ values often exceeding 0.99.
  • PDE Rediscovery: In complex tests, NOMTO-CNO rediscovered the 2D Heat Equation with an MSE of 0.004. For the Burgers' Equation, NOMTO-FNO correctly identified diffusion terms to the first decimal place.

Conclusion

NOMTO represents a significant step forward in automated scientific discovery. By integrating Neural Operators into the symbolic regression pipeline, it enables the discovery of complex differential equations that were previously out of reach for gradient-based methods. Future work will focus on improving the accuracy of NO surrogates and exploring advanced sparsity techniques to further refine the discovered equations.

Summary updated

BibTeX
@misc{garmaev2025nomto,
  author = {Sergei Garmaev and Siddhartha Mishra and Olga Fink},
  title = {{NOMTO: Neural Operator-based symbolic Model approximaTion and discOvery}},
  year = {2025},
  eprint = {2501.08086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.08086},
}
Figure