Abstract: Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam -- a multiplicative version of the Adam optimiser -- and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology.
Introduction
Deep learning models are inherently compositional, consisting of multiple layers stacked sequentially. While this structure allows for complex function approximation, it makes training with standard additive gradient descent ($W \leftarrow W - \eta \nabla L$) notoriously unstable. A minor adjustment in an early layer can propagate through the network, causing unpredictable shifts in the final output—a phenomenon often manifesting as vanishing or exploding gradients. Consequently, practitioners must engage in expensive hyperparameter searches to find the "perfect" learning rate for each specific task.
Furthermore, a significant efficiency gap exists between artificial and biological neural networks. While digital systems typically rely on 32-bit high-precision floating-point numbers, biological synapses are estimated to store only about 5 bits of information. This research addresses these challenges by proposing a multiplicative update rule that stabilizes training and enables high-performance learning with low-precision hardware.
Methods: Multiplicative Adam (Madam)
The core proposal is a shift from additive to multiplicative weight updates. The authors prove a "descent lemma" for compositional functions, demonstrating that updating weights by a relative factor rather than a fixed step size guarantees a decrease in loss without requiring knowledge of the network's complex second-order geometry (the Hessian).
Building on this, the authors introduce Madam, a multiplicative adaptation of the Adam optimizer. Instead of adding a momentum-based gradient, Madam scales the weight by an exponential factor:
$$W \leftarrow W \cdot \exp(-\eta \cdot \text{sign}(W) \cdot \text{clamped\_gradient})$$
Logarithmic Number Systems (LNS)
Multiplicative updates are naturally suited for Logarithmic Number Systems (LNS), where multiplications become simple additions in log-space. By representing weights using a sign bit and an exponent, the authors can quantize weights into a discrete "ladder" of values. This allows for significant compression, reducing the bit-width required for storage and computation.
Results and the "Universal" Learning Rate
The researchers benchmarked Madam against tuned SGD and Adam across diverse tasks, including image classification (CIFAR-10/100, ImageNet), generative modeling (cGAN), and language modeling (WikiText-2).
One of the most striking findings is the robustness of Madam to the learning rate parameter. While SGD and Adam require different optimal learning rates for different datasets, Madam achieved peak performance across nearly all tasks using a single "universal" learning rate of $\eta = 0.01$. This suggests that multiplicative updates are inherently more aware of the network's scale.
In terms of raw performance, Madam is highly competitive. On CIFAR-10, Madam reached a 7.8% error rate (compared to 6.6% for heavily tuned Adam). In generative tasks, Madam outperformed baselines, achieving an FID of 19 on a conditional GAN compared to 23.9 for standard optimizers. Furthermore, using 12-bit LNS weights, Madam performed almost identically to 32-bit floating-point models, demonstrating its efficacy for native model compression.
Biological Inspiration and Dale’s Principle
In biology, Dale’s Principle suggests that synapses are either excitatory or inhibitory and do not switch roles. Multiplicative updates naturally respect this constraint; because weights are scaled by a positive factor, the sign of the weight remains constant throughout training. This implies that the initial random sign pattern is sufficient for learning, mimicking the stability observed in biological systems.
Conclusion
Madam offers a promising alternative to standard optimizers by leveraging the compositional nature of neural networks. While it currently relies on weight clipping to prevent infinite growth and is limited by the initial sign pattern of weights, its stability and compatibility with low-precision logarithmic hardware mark a significant step toward more robust and efficient AI. Future work will focus on optimizing sign initialization and developing specialized hardware to fully realize the power-efficiency gains of logarithmic arithmetic.
Summary updated