Neural Power Units

Niklas Heim, Tomás Pevný, Václav Smídl

Introduction

Standard neural networks are powerful universal function approximators, but they are notoriously bad at extrapolation—generalizing to numerical ranges they haven't seen during training. For instance, an MLP trained to multiply numbers between 1 and 10 will likely fail when asked to multiply 100 by 100.

The Neural Power Unit (NPU) is a novel architecture designed to help neural networks perform arithmetic and learn power functions that generalize to numbers far outside their training range. By leveraging complex arithmetic and a specialized "relevance gate," the NPU overcomes the stability and domain limitations of previous arithmetic units, enabling models to "discover" underlying mathematical equations from raw data.

The Problem with Previous Arithmetic Units

Previous attempts to solve the extrapolation problem, such as the Neural Arithmetic Logic Unit (NALU), introduced inductive biases specifically for math. However, these units were plagued by two major issues:

  1. Domain Restrictions: They often couldn't handle negative numbers or division properly because they relied on real-valued logarithms ($\log |x|$).
  2. Convergence Fragility: Training was highly unstable. Specifically, if a single input in a product was near zero, the gradient for all other associated weights vanished, leading to "dead" units and failed training runs.

Methods: The Neural Power Unit (NPU)

The authors introduce the NPU, which represents arithmetic operations as products of power functions: $\prod x_i^{w_i}$.

NPU Architecture

1. Complex Arithmetic for Real Numbers

To handle negative inputs and arbitrary powers (like square roots or division), the NPU uses the complex logarithm. By treating weights and inputs as complex numbers internally, the unit can process the full range of real numbers $\mathbb{R}$. To keep the rest of the network standard, the authors use Euler’s formula to derive a version that only outputs the real part of the calculation, avoiding the need for complex-valued tensors throughout the entire model.

2. The Relevance Gate

The "Naive" NPU suffers from the zero-gradient problem: in a product, one zero makes the whole output zero. The authors solve this with a relevance gate ($g$).

Loss Landscapes with and without Relevance Gate

This gate learns to decide if an input is "relevant." If it isn't, the gate effectively turns that input into a $1$ (the identity element for multiplication). As shown in the loss landscapes above, this mechanism smooths the loss surface, preventing vanishing gradients and allowing the model to prune irrelevant features during training.

3. The RealNPU

For cases where interpretability is paramount, the authors propose the RealNPU—a simplified version where imaginary weights are removed. While slightly harder to train, it is highly "transparent," meaning researchers can look at the learned weights and directly read out a human-understandable mathematical formula.

Key Experiments and Results

The NPU was evaluated across several benchmarks, demonstrating significant improvements over existing methods:

  • Superior Extrapolation on Simple Arithmetic: When learning basic operations ($x+y, xy, x/y, \sqrt{x}$) on small ranges and testing on much larger ones, the NPU and RealNPU successfully extrapolated across the full range of real numbers. In contrast, the NALU and iNALU failed significantly, especially on division and square roots.
  • Solving Large-Scale Division: On a high-dimensional task requiring the model to sum subsets of a 100-dimensional vector and perform division, the NPU achieved a median testing error of $1.0 \times 10^{-7}$. The NALU's error was orders of magnitude higher ($530.0$).
  • Equation Discovery (fSIR Model): Using a simulated dataset based on a fractional-power Susceptible-Infectious-Recovered (SIR) model, the RealNPU successfully recovered the transmission rate formula $r = I^{0.62} S^{0.57}$ (where the ground truth was $I^{0.5} S^{0.5}$). It essentially "rediscovered" the underlying epidemiological laws from raw data.

Insights and Future Directions

Transparency and Inductive Bias: Unlike "black-box" MLPs, the NPU's weights correspond directly to exponents in a power function, allowing researchers to "read" the physics or logic the model has learned. By forcing the model to operate via power functions, the authors significantly reduce the number of parameters needed. In the ODE experiment, a handful of NPUs outperformed a much larger dense network.

Gating as a Regularizer: The relevance gate doesn't just help convergence; it encourages sparsity. It helps the model ignore noise and focus on the core variables driving the mathematical relationship.

Limitations: The highly interpretable RealNPU is more prone to getting stuck in local minima compared to the full NPU. The authors suggest that the imaginary parameters in the full NPU act as a "tunnel" that helps the optimizer avoid uninformative gradient regions. Additionally, the NPU requires specific initialization (Glorot for real weights, zeros for imaginary) to work effectively.

Future work may involve more sophisticated regularization of the imaginary parts to bridge the gap between the high convergence of the NPU and the high transparency of the RealNPU. There is also immense potential to integrate these units into broader "Universal Differential Equation" frameworks for scientific machine learning.

Summary updated

Abstract

Conventional Neural Networks can approximate simple arithmetic operations, but fail to generalize beyond the range of numbers that were seen during training. Neural Arithmetic Units aim to overcome this difficulty, but current arithmetic units are either limited to operate on positive numbers or can only represent a subset of arithmetic operations. We introduce the Neural Power Unit (NPU) that operates on the full domain of real numbers and is capable of learning arbitrary power functions in a single layer. The NPU thus fixes the shortcomings of existing arithmetic units and extends their expressivity. We achieve this by using complex arithmetic without requiring a conversion of the network to complex numbers. A simplification of the unit to the RealNPU yields a highly transparent model. We show that the NPUs outperform their competitors in terms of accuracy and sparsity on artificial arithmetic datasets, and that the RealNPU can discover the governing equations of a dynamical system only from data.
BibTeX
@inproceedings{heim2020neural,
  author = {Niklas Heim and Tomás Pevný and Václav Smídl},
  title = {{Neural Power Units}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=_WWxJ5vOVB},
}
Figure