Learning compositional functions via multiplicative weight updates

Jeremy Bernstein, Jiawei Zhao, Markus Meister, Ming-Yu Liu, Anima Anandkumar, Yisong Yue

In the world of deep learning, we’ve grown accustomed to the finicky nature of training. We spend countless GPU hours tuning learning rates to avoid the twin demons of vanishing and exploding gradients. But what if the problem isn't our tuning, but the very nature of our update rules?

A team of researchers recently presented a compelling alternative: Madam (Multiplicative Adam). By shifting from additive to multiplicative weight updates, they’ve created an optimizer that is not only more robust to hyperparameters but also bridges the gap between artificial neural networks and biological brains.

The Problem with Additive Updates

Standard optimizers like SGD and Adam use additive updates ($W \leftarrow W - \eta \nabla$). However, deep neural networks are compositional—they are essentially long chains of functions multiplied together. In such a system, a small additive change in an early layer can have an exponentially large (or small) effect on the final output. This is why deep learning is so sensitive to the learning rate.

Furthermore, there is a massive discrepancy between engineering and nature. Digital synapses typically use 32-bit precision (billions of values), while biological synapses are estimated to have only about 5 bits of precision (26 levels). Why does the brain learn so stably with such low precision, while our artificial models are so fragile?

Introducing Madam: Multiplicative Weight Updates

The authors propose a new mathematical framework based on "deep relative trust." They prove a descent lemma showing that for compositional functions, it is more principled to bound the relative change of weights rather than the absolute change.

This leads to the Madam algorithm, which features:

  • Multiplicative Updates: Weights are scaled by a factor (e.g., $W \times 1.01$) rather than having a value added to them.
  • Sign Preservation: Because the update is multiplicative, a weight never crosses zero. A positive weight stays positive; a negative one stays negative. This satisfies Dale’s Principle—the neuroscientific observation that synapses are exclusively excitatory or inhibitory and do not flip their sign during learning.
  • Logarithmic Number System (LNS): In hardware, multiplicative updates are expensive. However, if you store weights as logarithms, multiplication becomes simple addition. Madam is designed to run natively in this log-space.
13-bit Logarithmic Number System format used by Madam

To prevent weights from growing infinitely (a side effect of compound growth), the authors also implement a maximum weight clipping threshold.

The "One Learning Rate" Dream

Because layers are multiplied, the functional distance between two networks is better measured by the relative change in weights. Multiplicative updates respect this geometry.

Remarkably, the authors found that a single learning rate ($\eta = 0.01$) worked optimally across almost all tasks, from ResNets to Transformers to GANs. This suggests Madam could drastically reduce the need for hyperparameter searches.

Performance of Madam vs SGD and Adam across learning rates

As seen above, Madam's optimal learning rate aligns perfectly across diverse datasets (CIFAR, ImageNet, Wikitext-2), whereas SGD and Adam require distinct, carefully tuned learning rates for different tasks.

Key Results and Hardware Implications

The researchers benchmarked Madam against tuned versions of SGD and Adam across image classification, generative modeling, and language modeling:

  • Competitive Performance: Madam achieved results comparable to heavily tuned SGD and Adam. On ImageNet, Madam reached 28.9% error vs. SGD's 24.1%, but without the extensive tuning SGD requires.
  • GAN Stability: In the notoriously difficult task of training GANs, Madam significantly outperformed the baselines, achieving a much better FID score (19 vs. 23.9 for Adam; SGD failed to converge).
  • Low-Precision Success: Using a 12-bit logarithmic representation, Madam matched the performance of 32-bit floating-point models. Even at 8 bits, the model remained functional.

This low-precision success suggests that we don't necessarily need the complex floating-point units (FPUs) used in modern GPUs. A simpler "Logarithmic Number System" chip could be smaller, faster, and more power-efficient while maintaining training accuracy.

Limitations and Future Work

While Madam is a major step forward, it introduces a few new challenges. First, it requires tuning a new hyperparameter: the weight clipping threshold ($\sigma^*$). Second, because weights never change sign, the network is "locked into" the random sign pattern it receives at initialization, which could limit expressivity if the initial signs are poor. Finally, closing the slight performance gap on massive datasets like ImageNet remains a primary goal for future iterations.

Summary updated

Abstract

Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam---a multiplicative version of the Adam optimiser---and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology.
BibTeX
@inproceedings{bernstein2020learning,
  author = {Jeremy Bernstein and Jiawei Zhao and Markus Meister and Ming-Yu Liu and Anima Anandkumar and Yisong Yue},
  title = {{Learning compositional functions via multiplicative weight updates}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=WGtTbWCt-t},
}
Figure