Polynomial Neural Networks for Direct Symbolic Regression

19 concepts11 papers1 proposal

Standard deep neural networks are notoriously difficult to interpret. When we pass an input through layers of linear transformations interleaved with non-linear activation functions like ReLU or Sigmoid, the features become hopelessly entangled. We can observe the final output, but tracing exactly why a specific combination of inputs caused that output requires navigating a labyrinth of high-dimensional compositions.

To bridge this gap, a growing body of research replaces or approximates these complex black-box transformations with polynomial expansions. The core intuition is simple: a multivariate polynomial is inherently transparent. If a model's predictions can be expressed as a sum of individual variables and their products (e.g., $f(x) = 3x_1 + 0.5x_1x_2 - 2x_2^2$), we can directly read out the exact weight and significance of every feature interaction.

By structuring neural networks to output or mimic explicit polynomial forms, researchers are achieving direct symbolic regression—converting data-driven models into clean, executable mathematical equations without sacrificing the optimization advantages of gradient descent.


Designing Inherent Polynomial Architectures

One approach to achieving mathematical transparency is to build polynomial structures directly into the network's architecture. Instead of using standard activation functions, these networks utilize sum and product nodes or polynomial bases to ensure that the final learned function is natively algebraic.

Early foundations for this idea can be found in work by John Kassebaum et al., who introduced the Self Organizing Neural Network (The Cocktail Party Problem: Speech/Data Signal Separation Comparison between Backpropagation and SONN). By using second-order bivariate polynomials as prototype functions within the network's nodes, the authors demonstrated that a model could automatically construct its own architecture and be exported directly as a closed-form mathematical equation.

In the context of language processing, Kai Sun et al. developed the Recurrent Polynomial Network (RPN) for dialogue state tracking (Recurrent Polynomial Network for Dialogue State Tracking). By structuring state updates through explicit sum and product nodes, the network operates as a differentiable polynomial function. This allows domain experts to initialize the network with human-crafted logical rules and later inspect the fine-tuned weights to verify the model's decision-making logic. As shown by Qizhe Xie et al., this polynomial formulation provides a robust "middle ground" that maintains high performance even when the model encounters mismatched semantic parsers during deployment (Recurrent Polynomial Network for Dialogue State Tracking with Mismatched Semantic Parsers).

Modern iterations of this concept scale polynomial architectures to complex, high-dimensional tasks. For instance, Hongjue Zhao et al. proposed TaylorNet, an architecture that completely replaces traditional activation functions with Taylor polynomials parameterized by neural networks (TaylorNet: A Taylor-Driven Generic Neural Architecture). Similarly, Viet Duong et al. introduced Concept-bAsed Taylor additive models (CAT), which group raw features into high-level concepts and map them to outputs using transparent Taylor neural networks (CAT: Interpretable Concept-based Taylor Additive Models). Because the underlying mathematics relies on polynomial terms, the decision-making process can be explicitly visualized as shape functions, mapping out exactly how concepts interact to produce a prediction.

Even architectures inspired by alternative mathematical frameworks can be unified under this paradigm. Isha Puri et al. designed CoFrNets, a neural network architecture based on continued fractions (CoFrNets: Interpretable Neural Architecture Inspired by Continued Fractions). Because of their recursive structure, these networks can be mathematically unrolled and expanded into multivariate power series, exposing explicit feature interactions in a human-readable polynomial form.


Post-Hoc Expansions: Unrolling Black Boxes

Instead of restricting a network's architecture from the start, an alternative strategy allows a standard neural network to train normally and then converts its learned transformations into a polynomial representation post-hoc. This is typically achieved using high-order Taylor series or jet operators to locally or globally approximate the network's decision boundaries.

[Input Data] ---> [Standard Black-Box NN] ---> [Complex Output]
                         |
                         v (Taylor / Jet Expansion)
                  [Explicit Polynomial]  ---> [Symbolic Equations & Fast Inference]

Tingxiong Xiao et al. developed this concept with the HOPE (High-order Polynomial Expansion) framework (HOPE: High-order Polynomial Expansion of Black-box Neural Networks). By deriving efficient recurrence rules for high-order derivatives across standard network layers, the authors convert a trained model into an explicit Taylor polynomial around a reference point. This replaces thousands of sequential, layer-by-layer matrix multiplications with a single, highly parallelizable polynomial evaluation, yielding massive inference speedups alongside mathematical clarity.

This strategy has also been scaled to massive autoregressive models. Yihong Chen et al. introduced a framework that uses jet operators—a mathematical generalization of Taylor series—to decompose the computations of Large Language Models (Decomposing LLM Computation with Jets). This expansion separates the structured, polynomial part of the computation from a non-linear remainder term. By analyzing the resulting polynomial coefficients, researchers can extract symbolic $n$-gram tables and audit how information flows through the model's internal components directly from its weights.


Discovering Governing Physical Equations

The natural alignment between polynomials and physical laws makes this research direction exceptionally powerful for scientific discovery, particularly in modeling dynamical systems governed by differential equations.

In traditional scientific workflows, discovering a governing equation from data requires separate, multi-stage symbolic regression tools. However, by embedding polynomial networks inside differential equations, the model can learn the dynamics and the symbolic equation simultaneously. Colby Fronk demonstrated this by replacing the multilayer perceptron backbone of a Neural Ordinary Differential Equation (Neural NODE) with a deep polynomial neural network (Interpretable polynomial neural ordinary differential equations). Because the network's output layer is inherently polynomial, the underlying symbolic equations (such as the classical Lotka-Volterra or Van der Pol oscillators) can be directly extracted from the trained weights without any post-hoc symbolic fitting.

To make these discoveries reliable in noisy, real-world scenarios, Colby Fronk et al. introduced a Bayesian framework for these polynomial Neural NODEs (Bayesian polynomial neural networks and polynomial neural ordinary differential equations). By leveraging the Laplace approximation, this framework provides uncertainty quantification over the learned polynomial coefficients, allowing scientists to distinguish true physical signals from data noise.

[Noisy Trajectory Data] 
         |
         v
[Polynomial Neural NODE] --(Laplace Approximation)--> [Coefficients + Uncertainty Bounds]
         |
         v
[Extracted Symbolic Law]  (e.g., dx/dt = ax - bxy)

For partial differential equations (PDEs), Runzhao Yang et al. developed the SHoP framework (SHoP: A Deep Learning Framework for Solving High-Order Partial Differential Equations). By converting networks with infinitely differentiable activation functions into explicit Taylor series expansions, SHoP bypasses memory-intensive automatic differentiation. This mathematical conversion allows researchers to directly quantify the exact contributions of different input variables and verify that the network's solution strictly respects analytical physical constraints.

Finally, this paradigm extends to complex network structures. Riccardo Cappi et al. utilized Kolmogorov-Arnold Networks (KANs) to discover governing equations for graph dynamical systems (Discovering Generalizable Governing Equations for Graph Dynamical Systems with Interpretable Neural Networks). The inherent structure of KANs decomposes multivariate functions into combinations of univariate functions, providing a highly generalizable foundation for extracting explicit mathematical laws from networked systems.

Conclusion

Whether by constraining a network's architecture to polynomial primitives or unrolling trained black-boxes via Taylor expansions, this research direction fundamentally changes how we interact with deep learning models. By forcing neural computations into the transparent language of polynomials, these techniques turn deep networks into direct engines for symbolic regression, providing a clear path toward verifiable, interpretable, and mathematically explicit AI.

Key Foundational Work

The Cocktail Party Problem: Speech/Data Signal Separation Comparison between Backpropagation and SONN (1989) — contributed 1 concept elements to this direction.
Recurrent Polynomial Network for Dialogue State Tracking (2015) — contributed 1 concept elements to this direction.
Recurrent Polynomial Network for Dialogue State Tracking with Mismatched Semantic Parsers (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_direct_symbolic_regression_c4fb9539,
  title = {direct symbolic regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-symbolic-regression/rdr_c4fb9539a4e245cab22e66af02253c54},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_direct_symbolic_regression_c4fb9539_v1,
  title = {direct symbolic regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-symbolic-regression/rdr_c4fb9539a4e245cab22e66af02253c54/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake