Abstract
Conventional Neural Networks can approximate simple arithmetic operations, but fail to generalize beyond the range of numbers that were seen during training. Neural Arithmetic Units aim to overcome this difficulty, but current arithmetic units are either limited to operate on positive numbers or can only represent a subset of arithmetic operations. We introduce the Neural Power Unit (NPU) that operates on the full domain of real numbers and is capable of learning arbitrary power functions in a single layer. The NPU thus fixes the shortcomings of existing arithmetic units and extends their expressivity. We achieve this by using complex arithmetic without requiring a conversion of the network to complex numbers. A simplification of the unit to the RealNPU yields a highly transparent model. We show that the NPUs outperform their competitors in terms of accuracy and sparsity on artificial arithmetic datasets, and that the RealNPU can discover the governing equations of a dynamical system only from data.
Introduction
While deep learning has achieved remarkable success in domains like computer vision and natural language processing, it continues to struggle with fundamental arithmetic. A standard neural network trained to add numbers within a specific range (e.g., 1 to 10) typically fails to generalize when presented with inputs outside that range (e.g., 100 and 200). This lack of extrapolation—the ability to generalize beyond the training distribution—represents a significant barrier to using AI for scientific discovery and symbolic reasoning.
In their NeurIPS 2020 paper, Heim et al. introduced the Neural Power Unit (NPU). This specialized architectural layer is designed to learn complex arithmetic operations and arbitrary power functions across the entire real number line, providing a stable and transparent alternative to previous arithmetic units.
The NPU Architecture
Standard neural networks act as universal approximators within their training range but often fail to capture the underlying symbolic logic of a system. Previous attempts to address this, such as the Neural Arithmetic Logic Unit (NALU), suffered from two primary limitations: a reliance on logarithms that restricted them to positive inputs, and severe convergence issues due to vanishing gradients.
The NPU overcomes these hurdles through two key innovations:
- Complex Arithmetic Integration: To handle negative inputs, the NPU utilizes the complex logarithm. By performing internal calculations in the complex domain, the model preserves the sign of the data. To maintain compatibility with standard real-valued networks, the unit outputs only the real part of the final calculation, allowing it to represent functions like $y = x_1^{w_1} \cdot x_2^{w_2}$ for any real weight $w$.
- The Relevance Gate: In multiplicative units, a single zero input can zero out the entire product, effectively killing the gradient and halting learning. The NPU introduces a "relevance gate" that learns to ignore irrelevant inputs by smoothing them toward the identity value (1). This modification significantly improves the geometry of the loss landscape, making training more reliable.
Empirical Evaluation
The authors evaluated the NPU across several benchmarks, ranging from synthetic arithmetic to complex dynamical systems.
Simple and Large-Scale Arithmetic
On synthetic tasks requiring the simultaneous learning of addition, multiplication, division, and square roots, the NPU demonstrated superior extrapolation capabilities. While standard dense layers and NALUs failed when tested on inputs far outside the training range, the NPU maintained high accuracy. In high-dimensional tasks (e.g., operating on 100-dimensional vectors), the NPU achieved significantly lower Mean Squared Error (MSE) than its competitors while maintaining a sparser parameter count.
Equation Discovery in Dynamical Systems
A standout application of the NPU is its ability to discover governing equations from raw data. Using a simulated epidemiological dataset (the fSIR model), the RealNPU—a simplified version of the unit—successfully recovered the underlying differential equations. It correctly identified fractional powers (e.g., learning a weight of $\approx 0.6$ for a true power of $0.5$) and effectively ignored irrelevant variables through its gating mechanism.
Key Insights and Discussion
- Transparency by Design: Unlike traditional "black-box" layers, NPU weights have direct physical interpretations. A weight of $0.5$ assigned to an input $x$ explicitly signifies a square root relationship.
- Optimization via Complex Dimensions: The research suggests that complex weights provide additional dimensions for gradients to traverse, helping the model avoid the local minima that frequently plague real-valued arithmetic units.
- Limitations: While the NPU is more stable than its predecessors, the simplified RealNPU can still face convergence challenges. The model also requires careful initialization and $L_1$ regularization to ensure that the discovered equations remain sparse and human-readable.
Conclusion
The Neural Power Unit represents a significant step toward bridging the gap between deep learning and symbolic mathematics. By leveraging complex arithmetic and clever gating, it offers a path toward AI systems that don't just mimic patterns, but discover the mathematical laws governing the data. Future work may involve integrating NPUs into Universal Differential Equations to combine the flexibility of neural networks with the rigorous constraints of classical physics.
Summary updated