In the world of deep learning, we’ve grown accustomed to the finicky nature of training. We spend countless GPU hours tuning learning rates to avoid the twin demons of vanishing and exploding gradients. But what if the problem isn't our tuning, but the very nature of our update rules?
A team of researchers recently presented a compelling alternative: Madam (Multiplicative Adam). By shifting from additive to multiplicative weight updates, they’ve created an optimizer that is not only more robust to hyperparameters but also bridges the gap between artificial neural networks and biological brains.
The Problem with Additive Updates
Standard optimizers like SGD and Adam use additive updates ($W \leftarrow W - \eta \nabla$). However, deep neural networks are compositional—they are essentially long chains of functions multiplied together. In such a system, a small additive change in an early layer can have an exponentially large (or small) effect on the final output. This is why deep learning is so sensitive to the learning rate.
Furthermore, there is a massive discrepancy between engineering and nature. Digital synapses typically use 32-bit precision (billions of values), while biological synapses are estimated to have only about 5 bits of precision (26 levels). Why does the brain learn so stably with such low precision, while our artificial models are so fragile?
Introducing Madam: Multiplicative Weight Updates
The authors propose a new mathematical framework based on "deep relative trust." They prove a descent lemma showing that for compositional functions, it is more principled to bound the relative change of weights rather than the absolute change.
This leads to the Madam algorithm, which features:
- Multiplicative Updates: Weights are scaled by a factor (e.g., $W \times 1.01$) rather than having a value added to them.
- Sign Preservation: Because the update is multiplicative, a weight never crosses zero. A positive weight stays positive; a negative one stays negative. This satisfies Dale’s Principle—the neuroscientific observation that synapses are exclusively excitatory or inhibitory and do not flip their sign during learning.
- Logarithmic Number System (LNS): In hardware, multiplicative updates are expensive. However, if you store weights as logarithms, multiplication becomes simple addition. Madam is designed to run natively in this log-space.
To prevent weights from growing infinitely (a side effect of compound growth), the authors also implement a maximum weight clipping threshold.
The "One Learning Rate" Dream
Because layers are multiplied, the functional distance between two networks is better measured by the relative change in weights. Multiplicative updates respect this geometry.
Remarkably, the authors found that a single learning rate ($\eta = 0.01$) worked optimally across almost all tasks, from ResNets to Transformers to GANs. This suggests Madam could drastically reduce the need for hyperparameter searches.
As seen above, Madam's optimal learning rate aligns perfectly across diverse datasets (CIFAR, ImageNet, Wikitext-2), whereas SGD and Adam require distinct, carefully tuned learning rates for different tasks.
Key Results and Hardware Implications
The researchers benchmarked Madam against tuned versions of SGD and Adam across image classification, generative modeling, and language modeling:
- Competitive Performance: Madam achieved results comparable to heavily tuned SGD and Adam. On ImageNet, Madam reached 28.9% error vs. SGD's 24.1%, but without the extensive tuning SGD requires.
- GAN Stability: In the notoriously difficult task of training GANs, Madam significantly outperformed the baselines, achieving a much better FID score (19 vs. 23.9 for Adam; SGD failed to converge).
- Low-Precision Success: Using a 12-bit logarithmic representation, Madam matched the performance of 32-bit floating-point models. Even at 8 bits, the model remained functional.
This low-precision success suggests that we don't necessarily need the complex floating-point units (FPUs) used in modern GPUs. A simpler "Logarithmic Number System" chip could be smaller, faster, and more power-efficient while maintaining training accuracy.
Limitations and Future Work
While Madam is a major step forward, it introduces a few new challenges. First, it requires tuning a new hyperparameter: the weight clipping threshold ($\sigma^*$). Second, because weights never change sign, the network is "locked into" the random sign pattern it receives at initialization, which could limit expressivity if the initial signs are poor. Finally, closing the slight performance gap on massive datasets like ImageNet remains a primary goal for future iterations.
Summary updated