Deep Polynomial Neural Networks via Multiplicative Interactions

40 concepts30 papers1 proposal

Standard deep learning architectures rely on a familiar recipe: alternate linear transformations ($Wx + b$) with element-wise, non-linear activation functions ($\sigma$) like ReLU or GeLU. While this paradigm is highly effective, it forces networks to approximate complex, high-order feature interactions implicitly through depth. For instance, to capture the joint interaction between two features $x_1$ and $x_2$, a standard network must use multiple layers of piecewise-linear splits to approximate the product $x_1 x_2$.

An alternative paradigm shifts the source of non-linearity from scalar-wise activations to multiplicative feature interactions. By replacing or augmenting standard additive layers with element-wise multiplications (Hadamard products) or multilinear operators, these networks directly construct high-order polynomial expansions of their inputs. This architectural shift creates models that are highly expressive, parameter-efficient, and capable of operating entirely without traditional activation functions.

The Core Mechanism: Multiplication as Non-Linearity

To understand why multiplicative interactions can replace traditional activations, consider two distinct linear transformations of an input vector $x$:

$$A(x) = W_1 x \quad \text{and} \quad B(x) = W_2 x$$

If we compute their element-wise product (the Hadamard product), we get:

$$f(x) = A(x) \odot B(x)$$

The resulting vector $f(x)$ contains quadratic terms ($x_i x_j$). This operation is inherently non-linear, yet it requires no activation function.

When this mechanism is applied recursively across layers, the degree of the polynomial grows exponentially. If Layer 1 produces a degree-2 polynomial, multiplying its output with another linear transformation yields a degree-3 or degree-4 polynomial. As shown by Chrysos et al. in their work on $\Pi$-nets, recursively applying element-wise multiplication to feature maps allows a network to output a high-order polynomial of the input signal.

Standard Layer:      Input (x) ---> [ Linear Layer ] ---> [ ReLU / GeLU ] ---> Output
Multiplicative:      Input (x) ---> [ Linear A ] ---\
                                                     ---> [ Element-wise ] ---> Output
                                    [ Linear B ] ---/       Multiplication

By stacking $L$ of these multilinear modules, a network can capture $2^L$-order feature interactions, as demonstrated by ChangNie. This hierarchical composition allows the model to approximate highly complex functions using only addition and multiplication (Cheng et al.), bypassing the piecewise-linear constraints of standard architectures.

Why It Works: Explicit Feature Crossing

In many real-world tasks, the underlying data relationships are explicitly polynomial. In click-through rate (CTR) prediction, for example, a user's behavior often depends on the conjunction of categorical features (e.g., User Country == USA AND Language == English).

Standard deep networks require an excessive number of parameters to approximate these simple polynomial crossings. To solve this, the Deep & Cross Network proposed by Wang et al. introduces specialized layers that calculate the product of the original input and the current hidden state. This systematically increases the degree of polynomial interactions with each additional layer while maintaining a clear mathematical structure. Similarly, Li et al. demonstrate that the inherent ability of the Hadamard product to expand the feature space provides smooth, non-linear approximations that make traditional activation functions less critical.

This explicit modeling is also powerful in physical systems and generative tasks:

  • Physical Systems: In fluid dynamics and chaotic systems, the governing laws are often nonlinear differential equations. Rao et al. utilize multiplicative feature blocks to approximate these nonlinear polynomial terms directly from scarce data.
  • Conditional Generation: Chrysos developed CoPE, which uses a recursive multivariate polynomial structure to model the interaction between latent noise and conditional inputs (like class labels), capturing intricate cross-correlations layer by layer.

In fact, this perspective unifies many popular architectures. Chrysos et al. showed that standard components like ResNet skip connections and gating mechanisms (e.g., SENet) can be viewed as truncated polynomial expansions. By identifying and adding the "missing degrees" to the feature interaction hierarchy, they created Polynomial Deep Classifiers that achieve superior accuracy with up to 60% fewer parameters.

Taming the Combinatorial Explosion

The primary challenge of polynomial expansion is the curse of dimensionality. The number of terms in a full polynomial expansion grows combinatorially with the degree $d$ and the number of features $n$:

$$\text{Number of terms} = \binom{n+d}{d}$$

For a model with thousands of features, computing a high-order polynomial directly is computationally impossible. Polynomial networks solve this through tensor factorizations and recursive formulations.

Instead of learning a separate weight for every possible feature combination, models constrain the interaction weights to a low-rank space. For example, Exponential Machines (Novikov et al.) utilize the Tensor Train format to represent an exponentially large weight tensor compactly. This allows the model to capture all orders of feature interactions while scaling linearly with the number of features.

Similarly, the authors of Deep Polynomial Neural Networks employ coupled tensor factorizations to prevent parameter explosion, ensuring the network remains highly parameter-efficient. In statistical contexts, Rügamer leverages a recursive formulation to efficiently compute symmetric polynomials of basis functions, capturing multi-way dependencies without ever explicitly calculating the expensive tensor products.

Practical Implications: Hardware and Privacy

By replacing transcendental functions (like Sigmoid or Tanh) and non-differentiable piecewise functions (like ReLU) with pure algebraic operations (addition and multiplication), polynomial networks unlock unique engineering advantages:

  1. Privacy-Preserving Machine Learning: Fully Homomorphic Encryption (FHE) allows processing encrypted data, but it natively supports only addition and multiplication. Traditional activation functions are incredibly costly to compute under FHE because they must be approximated using polynomials. Multilinear Operator Networks (Cheng et al.) eliminate this overhead entirely by being polynomial by design, making them natively compatible with secure computing.
  2. Alternative Computing Hardware: Non-linear processing in optical computing typically requires high-power pulsed lasers to trigger material non-linearities. However, Yildirim et al. demonstrate that repetitively embedding input data across multiple scattering layers allows linear optical systems to synthesize high-order polynomial transformations using low-power, continuous-wave light.
  3. Efficient Attention Mechanisms: Standard self-attention scales quadratically $O(N^2)$ with sequence length. Letourneau introduces PADRe, showing that replacing the softmax attention matrix with hardware-friendly polynomial functions achieves linear complexity, revealing that many modern sub-quadratic architectures are instances of this underlying polynomial mathematics.

Key Foundational Work

Coupled Interdependent Attribute Analysis on Mixed Data (2015) — contributed 1 concept elements to this direction.
A Neural Probabilistic Structured-Prediction Model for Transition-Based Dependency Parsing (2015) — contributed 1 concept elements to this direction.
Training log-linear acoustic models in higher-order polynomial feature space for speech recognition (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multiplicative_non_linearity_11467c75,
  title = {multiplicative non-linearity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multiplicative-non-linearity/rdr_11467c7514c14f349094f61541b6f18b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multiplicative_non_linearity_11467c75_v1,
  title = {multiplicative non-linearity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multiplicative-non-linearity/rdr_11467c7514c14f349094f61541b6f18b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake