Interpretable Neural Architectures via Explicit Polynomial Expansions

This direction is split into other directions: direct symbolic regression. Its last version stays readable and citable here.

Version 1 of 120 papers28 conceptsfrozen 2026-10-04

Neural networks are often criticized as "black boxes" because their internal logic is buried within millions of non-linear transformations. However, a growing body of research suggests that we can crack these boxes open by treating neural networks as high-order polynomials. By either designing architectures to be inherently polynomial or "unrolling" existing models into Taylor series, researchers are transforming opaque weights into human-readable mathematical expressions.

The Intuition: From Neurons to Taylor Series

At its core, a standard neural network is a composition of linear layers and non-linear activation functions like ReLU or Sigmoid. While these functions are powerful, they are mathematically "messy" to interpret globally.

The central insight of this research direction is that any sufficiently smooth function can be approximated by a Taylor series—a sum of polynomial terms (like $x$, $x^2$, $x_1x_2$, etc.). If we can represent a neural network as an explicit polynomial, the "black box" disappears. We are left with a list of features and their interactions, where the coefficients tell us exactly how much each interaction contributes to the final prediction.

Unrolling the Black Box

One approach is to take a pre-trained, standard neural network and "unroll" it into a polynomial form. Yang Li demonstrated that because ReLU-based networks act as local linear splines, they can be used to estimate higher-order derivatives and reconstruct the model’s logic as a Taylor polynomial.

This concept has been scaled significantly by Tingxiong Xiao and colleagues. In their work on HOPE (High-order Polynomial Expansion), they developed recurrence rules to efficiently calculate high-order derivatives across network layers. This allows them to convert a complex model into a single, transparent polynomial at a specific reference point. This isn't just for interpretation; these polynomial surrogates can perform inference up to 3,700 times faster than the original models because they replace deep layer-by-layer traversal with a single mathematical evaluation.

More recently, Yihong Chen and Luca Franceschi introduced Jet Expansions, which use "jets" (a generalization of Taylor series) to decompose the entangled computations of Large Language Models. By unrolling these computations into explicit paths, they can extract symbolic $n$-gram tables directly from a model's weights, revealing the specific linguistic patterns the model has learned.

Designing for Polynomial Consistency

Rather than converting models after training, another group of researchers builds networks that are inherently polynomial. This is particularly vital in the physical sciences, where the laws of nature (like fluid dynamics or quantum mechanics) are often expressed as polynomial interactions between variables.

Standard neural networks struggle with these because they are primarily additive (summing weighted inputs). To capture the multiplicative nature of physics, researchers like Chengping Rao and Hao Sun developed the PeRCNN architecture. It uses specialized "$\Pi$-blocks" (multiplicative layers) that allow the network to naturally represent the nonlinear terms found in partial differential equations (PDEs).

Similarly, Maarten V. de Hoop and Christopher A. Wong proposed integrating input data multiplicatively into every layer. This allows the network to represent matrix polynomials, which are the fundamental language of operator-based physical equations.

Why It Works: Bridging Physics and Deep Learning

The reason these polynomial architectures are so effective in scientific domains is that they mirror the structure of classical numerical analysis.

  1. Physical Observables: In quantum matter research, Cole Miles and Annabelle Bohrdt developed Correlator Convolutional Neural Networks (CCNNs). By replacing standard activations with polynomial structures, they ensure the learned features correspond exactly to multi-point correlation functions—physical quantities that scientists already know how to measure.
  2. Equation Discovery: Because the weights of these networks correspond to coefficients in a polynomial, we can "read" the physics back out. Kevin Zhang used this to identify mysterious phases of quantum matter, translating network weights into explicit mathematical expressions for multi-body interactions.
  3. Mathematical Transparency: Even outside of pure physics, architectures like CoFrNets, introduced by Amit Dhurandhar, use continued fractions to maintain the power of universal approximators while remaining expandable into power series. This provides a mathematically grounded way to identify which feature interactions are driving a model's global behavior.

By treating the "neurons" of a network not as abstract biological metaphors but as terms in a high-order polynomial, this research direction is turning deep learning into a rigorous, interpretable extension of classical mathematics.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_interpretable_neural_architectures_via_e_507c78b3,
  title = {Interpretable Neural Architectures via Explicit Polynomial Expansions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interpretable-neural-architectures-via-explicit-polynomial-expansions/rdr_507c78b3c28c43989273321584f8649a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_interpretable_neural_architectures_via_e_507c78b3_v1,
  title = {Interpretable Neural Architectures via Explicit Polynomial Expansions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interpretable-neural-architectures-via-explicit-polynomial-expansions/rdr_507c78b3c28c43989273321584f8649a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}