In the physical sciences, we often face a trade-off between two types of models. On one hand, we have neural networks: high-capacity "black boxes" that can fit almost any data but offer little insight into the underlying laws. On the other, we have symbolic equations: concise, human-readable formulas like $F=ma$ that generalize beautifully but are notoriously difficult to "discover" from raw, noisy data.
A growing body of research suggests we don't have to choose. By using neural networks as an intermediate "bridge," researchers are now able to distill complex machine learning models into the elegant mathematical language of physics.
The Combinatorial Explosion of Symbols
The traditional way to find a formula for a dataset is Symbolic Regression (SR). SR searches through a massive library of mathematical operators ($+$, $\sin$, $\exp$, etc.) to find the simplest expression that fits the data.
However, SR has a major weakness: it scales poorly. As the number of variables increases, the search space of possible equations explodes combinatorially. Furthermore, SR is highly sensitive to noise; a few outliers can lead the algorithm to propose a "correct" but nonsensical formula with dozens of nested terms.
The Neural Surrogate as a Denoising Bridge
To solve this, researchers like Miles Cranmer, Shirley Ho, and Peter W. Battaglia proposed a two-step pipeline in Learning Symbolic Physics with Graph Networks.
Instead of running symbolic regression on raw data, they first train a Graph Neural Network (GNN) to model the system. The GNN is designed with a "physical" inductive bias: it learns to pass "messages" between nodes (particles). Once the GNN is trained, the researchers apply symbolic regression only to the internal functions the network learned—specifically the message-passing function.
Why does this work? The neural network acts as a high-fidelity surrogate. It smooths out the noise and interpolates the sparse data, providing a "clean" numerical target for the symbolic search. This allows SR to focus on a small, specific sub-problem (like the force between two particles) rather than trying to solve the entire system at once. This approach has even been used to rediscover orbital mechanics from NASA trajectory data.
Filling the Gaps in Known Physics
Sometimes we know most of the physics but are missing a specific term—for example, a complex friction force or a biological growth rate. Christopher Rackauckas and colleagues introduced Universal Differential Equations (UDEs) to address this in Universal Differential Equations for Scientific Machine Learning.
In a UDE, a neural network is embedded directly inside a differential equation to represent the unknown part of the dynamics. After training the system to match observations, the neural network's output is distilled into a symbolic formula. This "hybrid" approach ensures the model stays grounded in known physical laws while using the neural network to "discover" the missing pieces, such as the governing equations for chaotic neural populations.
Architectures for Intrinsic Interpretability
While the two-step "train then distill" pipeline is powerful, a newer direction seeks to build interpretability directly into the network's architecture. Kolmogorov-Arnold Networks (KANs), and their operator-based extensions like KANO by Ziming Liu and others, replace fixed activation functions with learnable splines on the edges of the network.
Because these splines are univariate functions, they can be easily "symbolified" after training. A researcher can look at a trained KAN, see that a specific edge has learned a shape resembling a sine wave, and replace it with a symbolic $\sin(x)$ term. This process of sparsification and pruning transforms the opaque weights of a network into a transparent, executable formula.
Why Distillation Matters
The goal of this research is not just to replace neural networks, but to use them as a tool for scientific insight. By distilling a model into a symbolic form, we gain several advantages:
- Generalization: Symbolic formulas often generalize to regimes far outside the training data where neural networks typically fail.
- Efficiency: A simple algebraic equation can be evaluated on low-power hardware or integrated into traditional solvers with near-zero overhead.
- Trust: In fields like healthcare or industrial maintenance, "white-box" formulas allow experts to audit the model's logic before making critical decisions.
Ultimately, symbolic distillation treats deep learning as a powerful "pre-processor" for the human mind, turning the chaos of big data into the clarity of scientific law.