Distilling Neural Networks into Closed Form Expressions

566 concepts367 papers3 proposals

Deep neural networks are exceptional at finding patterns in high-dimensional data, but they suffer from two fundamental flaws when applied to scientific discovery: they are black boxes whose internal logic cannot be easily audited, and they struggle to extrapolate reliably outside their training distribution. In contrast, classical physics relies on concise, closed-form mathematical equations. These symbolic expressions are inherently interpretable and generalize perfectly because they capture the underlying causal mechanisms rather than superficial statistical correlations.

The challenge is that searching the space of all possible mathematical expressions—combining variables, constants, and operators like addition, sine, or logarithms—is a discrete, combinatorial optimization problem. Brute-force search quickly explodes. To solve this, a powerful paradigm has emerged: using the continuous optimization power of neural networks to find the right functional landscape, and then distilling those networks into exact symbolic equations.

Two-Stage Distillation: Learn Then Read Out

One of the most intuitive ways to bridge this gap is a two-stage workflow: first, train a high-capacity neural network to model the system, and second, use symbolic regression to extract the mathematical laws hidden within its weights or activations.

For instance, in physical systems where geometry and interactions matter, graph neural networks (GNNs) can be trained to predict complex bulk behaviors. Ruoxia Chen demonstrated that a GNN can first learn the energy mappings of disordered atomic systems, after which symbolic regression can distill the network's internal representations back into precise analytical formulas for interatomic forces. Similarly, Suresh Bishnoi et al. developed Hamiltonian Graph Neural Networks that explicitly decouple kinetic and potential energy from raw trajectories in Discovering symbolic laws directly from trajectories with hamiltonian graph neural networks. By applying symbolic regression to individual neural modules within this architecture, the authors successfully "read out" exact classical laws like the Lennard-Jones potential.

This strategy also works by targeting specific internal bottlenecks. Sebastian Johann Wetzel et al. utilized Siamese Neural Networks with a low-dimensional bottleneck layer to determine if two states belong to the same physical system in Discovering symmetry invariants and conserved quantities by interpreting Siamese Neural Networks. By performing polynomial regression directly on the activations of the bottleneck neuron, they translated the learned representations into explicit conservation laws. When neural networks are trained to preserve invariants, as shown by Takashi Matsubara et al. with the FINDE framework, symbolic regression can easily convert those invariant-preserving functions into human-readable equations for energy or momentum.

Instead of treating the neural network and the symbolic search as entirely separate steps, another approach makes the symbolic search itself differentiable by relaxing discrete mathematical structures into continuous spaces.

Ahmed Alaa et al. addressed this in Demystifying Black-box Models with Symbolic Metamodels by introducing symbolic metamodels based on Meijer G-functions. Because Meijer G-functions encompass a massive class of familiar elementary and transcendental functions, they allow the discrete task of searching for a functional form to be rewritten as a continuous optimization problem solvable via gradient descent.

Another approach is to relax the structure of programs or equations. Ameesh Shah et al. developed the NEAR framework in Learning Differentiable Programs with Admissible Neural Heuristics, which frames symbolic program synthesis as a graph traversal problem. By placing neural networks as differentiable "placeholders" for missing blocks of code, they create an admissible heuristic that guides discrete search algorithms like A* toward optimal, compact program architectures. Alternatively, reinforcement learning can be used to navigate the discrete tree space of equations. Brenden K. Petersen et al. utilized a recurrent neural network acting as an agent that generates mathematical expressions, optimized via a risk-seeking policy gradient that prioritizes best-case performance to find exact symbolic formulas.

Natively Symbolic and Interpretable Architectures

A third strategy eliminates the separate distillation step by embedding symbolic constraints directly into the neural network architecture itself, forcing the model to learn via interpretable building blocks from the start.

Subham S. Sahoo et al. introduced EQL÷ in Learning Equations for Extrapolation and Control, a neural network architecture where standard activation functions are replaced by basic mathematical operators, including a specialized division unit. Because the network is structurally restricted to these primitive operations, sparsifying the weights automatically yields a concise, human-readable equation that can extrapolate perfectly to unseen data regions. In a similar vein, Zichao Long et al. designed PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network, a hybrid network that combines learnable convolutional filters (representing differential operators) with symbolic layers to discover the exact algebraic forms of partial differential equations from noisy spatial data.

Recently, Kolmogorov-Arnold Networks (KANs) have emerged as a powerful alternative to multi-layer perceptrons by placing learnable univariate spline functions on the edges rather than fixed activations on the nodes. As shown by W. M. Liu et al. in Lightweight posterior construction for gravitational-wave catalogs with the Kolmogorov-Arnold network, this allows for "symbolification"—matching the trained edge splines to elementary functions like exponentials or polynomials to compress massive data distributions into compact analytic expressions. Kota Yoshida et al. demonstrated in Can KANs Do It? Toward Interpretable Deep Learning-based Side-channel Analysis that this mechanism allows researchers to extract explicit computational logic, such as periodic modular arithmetic, directly from the network's weights.

Why It Works: The Synergy of Smoothness and Sparsity

Why does combining neural networks with symbolic regression work so much better than running a discrete search directly on raw data?

The answer lies in how neural networks reshape the optimization landscape. Raw experimental data is often noisy, sparse, and plagued by high-dimensional correlations. Running a pure genetic algorithm or sparse regression dictionary (like PySINDy by Brian M. de Silva et al.) directly on raw data can fail if the numerical derivatives are noisy or if the true functional form is missing from a predefined library.

When a neural network is trained first—especially one regularized by physics-informed losses, as seen in the work of Lena Podina et al.—the network acts as a continuous, noise-filtering surrogate. It smooths out the data, interpolates missing regions, and provides clean, analytical gradients. When the symbolic regression algorithm subsequently fits the neural network's predictions or internal activations, it operates on a clean, idealized mathematical manifold rather than a noisy cloud of data points. By combining the global, continuous optimization of deep learning with the discrete, parsimonious selection of symbolic regression, researchers can reliably extract the exact, elegant mathematical laws governing complex systems.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 375 papers
  2. Version 1 · 2026-10-04 · 197 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_symbolic_regression_91533e13,
  title = {symbolic regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symbolic-regression/rdr_91533e138824448fa8a9cad639b9c542},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_symbolic_regression_91533e13_v2,
  title = {symbolic regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symbolic-regression/rdr_91533e138824448fa8a9cad639b9c542/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake