How Do Adam and Training Strategies Help BNNs Optimization

Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, Kwang-Ting Cheng

Introduction

Binary Neural Networks (BNNs) represent a "holy grail" for edge computing. By constraining both weights and activations to a single bit (-1 or +1), BNNs drastically reduce memory footprints and enable acceleration via simple logic gates on hardware. However, this extreme quantization makes training notoriously difficult.

In the realm of traditional, real-valued deep neural networks, Stochastic Gradient Descent (SGD) with momentum is the gold standard due to its superior generalization capabilities compared to adaptive methods like Adam. Yet, researchers have consistently observed a paradox in BNNs: Adam significantly outperforms SGD. This paper investigates the chaotic, rugged loss landscapes of BNNs to uncover exactly why Adam succeeds where SGD fails, and leverages these insights to push the boundaries of BNN performance.

The Challenge: Chaotic Landscapes and "Dead" Weights

To understand the optimization process, the authors visualized the loss surfaces of both real-valued and binary networks. They found that BNN landscapes are highly chaotic, riddled with numerous local minima that trap standard optimizers.

A primary culprit behind training failure in BNNs is activation saturation. During training, if an activation falls outside the effective [-1, 1] range of the sign function, its gradient drops to zero. When using SGD, a weight with a zero gradient stops updating entirely, effectively becoming "dead."

Why Adam Prevails

The study reveals that Adam's unique mechanics naturally counteract the pitfalls of BNN optimization:

  • Revitalizing Dead Weights: Adam’s second momentum tracks the moving average of squared gradients, acting as a normalizer. When gradients shrink to near-zero due to saturation, Adam effectively scales up the update step. This adaptive learning rate "kicks" the dead weight back into a functional, active range.
  • Filtering Gradient Noise: BNN optimization is inherently noisy due to the discrete nature of the weights. Comparisons with other adaptive optimizers, such as AdaGrad and RMSProp, demonstrated that Adam’s first momentum is equally critical, as it smooths out erratic gradient fluctuations.

Refining BNN Training Strategies

Beyond the choice of optimizer, the authors introduce new metrics and strategies to stabilize training, focusing on the behavior of "latent weights." Even though BNNs use 1-bit weights during the forward pass, they maintain high-precision, real-valued latent weights during training to accumulate small gradient updates. The magnitude of a latent weight represents the model's "confidence" in its binary sign.

To optimize these latent weights, the authors explored two key areas:

  • The Weight Decay Dilemma: In standard networks, weight decay prevents overfitting. In BNNs, it controls the Flip-Flop (FF) Ratio (how often binary weights switch signs) and the Correlation-to-Initialization (C2I) Ratio (how much final weights depend on their random starting values). High weight decay forces latent weights near zero, causing them to flip-flop unstably. Zero weight decay leaves them too dependent on initialization. The authors identified a "sweet spot" (5e-6) that perfectly balances stability and flexibility.
  • Two-Step Training (BABW): The sequence of binarization matters. The authors found that binarizing activations before weights (BABW) is vastly superior to the reverse. This approach allows weights to remain real-valued longer, reducing initialization dependency before they are locked into discrete 1-bit values.

State-of-the-Art Results

By applying these insights to the ReActNet-A architecture and training on the large-scale ImageNet 2012 dataset, the authors achieved a record-breaking 70.5% top-1 accuracy. This marks a 1.1% improvement over the previous state-of-the-art, crossing a major psychological threshold for 1-bit networks.

Furthermore, on a binary ResNet-18, Adam achieved 61.49% accuracy compared to SGD’s 58.98%, while proving significantly more robust to the choice of initial learning rate.

Conclusion and Future Directions

While Adam is currently the best tool for training BNNs, it was not originally designed for them. The training process still relies heavily on high-precision latent weights, meaning a truly "binary-from-scratch" training method remains elusive. The authors conclude that future research should focus on designing entirely new optimizers explicitly tailored for the discrete, rugged optimization landscapes of 1-bit quantization.

Summary updated

Abstract

The best performing Binary Neural Networks (BNNs) are usually attained using Adam optimization and its multi-step training variants. However, to the best of our knowledge, few studies explore the f...
BibTeX
@inproceedings{liu2021how,
  author = {Zechun Liu and Zhiqiang Shen and Shichao Li and Koen Helwegen and Dong Huang and Kwang-Ting Cheng},
  title = {{How Do Adam and Training Strategies Help BNNs Optimization}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure