Deep Learning with Quadratic Neurons

22 concepts13 papers1 proposal

Traditional artificial neurons compute a linear combination of their inputs followed by a non-linear activation function:

$$\text{Output} = \sigma\left(\sum_i w_i x_i + b\right)$$

While stacking these layers deeply allows neural networks to learn incredibly complex functions, an individual conventional neuron is fundamentally blind to feature interactions. It cannot compute simple multiplicative relationships—such as an XOR gate or a feature covariance—without relying on subsequent layers to assemble those relationships.

Quadratic neurons change this paradigm by replacing the standard linear summation with a quadratic function. Inspired by biological dendritic integration, where individual neurons explicitly calculate interactions between incoming signals rather than just summing them (Liu et al., 2024), quadratic neurons allow a single processing unit to capture second-order feature correlations directly.

The Mechanism: Efficient Second-Order Interactions

A naive implementation of a quadratic neuron would require computing a full matrix product $x^T V x$, where $V$ is a weight matrix. For a $d$-dimensional input, this incurs an $O(d^2)$ computational and parameter overhead, making it prohibitively expensive for modern deep learning architectures.

To make quadratic networks practical, researchers leverage low-rank factorizations and element-wise operations. A common and highly efficient formulation uses the Hadamard product ($\odot$) of two distinct linear projections. Instead of computing a full quadratic form, the core interaction is modeled as:

$$\text{Output} = \sigma\left( (W_1 x) \odot (W_2 x) \right)$$

As demonstrated by Bu et al. (2021) and Xu et al. (2022), this formulation captures second-order polynomial spaces while maintaining linear computational complexity relative to the input size. To optimize this further, Chen et al. (2025) introduced methods to reuse the primary linear output as the basis for the higher-order term, multiplying the primary linear output by a transformed version of itself to gain expressiveness with virtually zero computational overhead.

Why It Works: Expressivity and Vertical Compression

Modifying the basic building block of neural networks yields significant advantages in parametric efficiency and function approximation theory.

Traditional ReLU networks approximate functions using piecewise linear boundaries. In contrast, quadratic neurons inherently represent piecewise polynomial functions, unlocking a significantly richer function space (Wang et al., 2025). Theoretical analyses by Fan et al. (2020) and Fan et al. (2025) demonstrate that quadratic networks can approximate complex function classes—such as radial functions or functions within Sobolev and Barron spaces—with a polynomial number of parameters, whereas conventional networks often require an exponential number of neurons to achieve the same accuracy.

Standard Neuron:  [Inputs] ---> [Linear Summation] ---> [Activation] ---> Piecewise Linear Boundary

Quadratic Neuron: [Inputs] ---> [Second-Order Interaction] -> [Activation] ---> Piecewise Quadratic Boundary

This mathematical efficiency enables vertical architecture contraction. Because individual layers are inherently more expressive, shallower quadratic networks can match or exceed the performance of much deeper conventional architectures (Xu et al., 2022). Furthermore, modeling learning dynamics via quadratic expansions reveals that second-order interactions capture non-convex feature learning that standard linear approximations, like the Neural Tangent Kernel (NTK), completely overlook (Li et al., 2019; Chen et al., 2020).

Bypassing Heavy Global Mechanisms

By embedding feature interactions directly into the neuron, quadratic architectures can bypass expensive global coordination mechanisms like self-attention, message passing, or dense multi-layer feature crossing.

  • Replacing Self-Attention: In Transformers, self-attention computes global token-to-token interactions, which scales quadratically with sequence length. Xu et al. (2023) demonstrated that replacing self-attention mechanisms with localized quadratic neurons captures high-order feature interactions efficiently, maintaining Transformer-level accuracy while reducing memory footprint and increasing throughput.
  • Eliminating Message Passing: In graph-based brain network modeling, complex message-passing schemes are standard. However, Yang et al. (2025) showed that a Brain Quadratic Network (BQN) utilizing Hadamard products can implicitly model community structures and intricate dependencies without explicit message passing.
  • Streamlining Feature Interactions: In click-through rate (CTR) prediction, explicitly modeling feature interactions is crucial. Li et al. (2025) framed traditional Hadamard-product interactions within the formal context of Quadratic Neural Networks, showing that explicit quadratic terms expand the feature space smoothly and reduce the network's reliance on deep activation stacks for core interaction modeling. Similarly, Li et al. (2023) utilized explicit product calculations between pairs of neurons to capture complex lateral and feedback interactions, helping networks navigate complex loss landscapes.

Stabilizing the Training of Quadratic Neurons

Despite their theoretical advantages, pure quadratic functions are notoriously difficult to train due to gradient explosion and optimization instability. Because the output scales quadratically with the weights and inputs, the loss landscape can become highly non-convex and erratic.

To mitigate this, modern quadratic architectures incorporate a stabilizing linear term alongside the quadratic component. For example, the architecture proposed by Xu et al. (2022) combines a Hadamard-product quadratic term with a parallel linear projection:

$$\text{Output} = \sigma\left( (W_1 x \odot W_2 x) + W_3 x \right)$$

The parallel linear term ensures a stable gradient flow during the early stages of training, preventing the vanishing or exploding gradients typical of pure higher-order polynomials. Additionally, the authors of other recent studies have developed specific initialization and regularization strategies—such as the "ReLinear" strategy proposed by Wang et al. (2025)—designed to tame these optimization dynamics, making quadratic networks as robust and trainable as standard deep neural networks.

Key Foundational Work

Universal approximation with quadratic deep networks (2020) — contributed 3 concept elements to this direction.
Towards Understanding Hierarchical Learning: Benefits of Neural Representations (2020) — contributed 1 concept elements to this direction.
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_quadratic_neurons_f1f0005c,
  title = {quadratic neurons},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quadratic-neurons/rdr_f1f0005c43c643949d16e368f15897e8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_quadratic_neurons_f1f0005c_v1,
  title = {quadratic neurons},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quadratic-neurons/rdr_f1f0005c43c643949d16e368f15897e8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake