Abstract
Binary Neural Networks (BNNs) offer significant computational and storage advantages for edge AI but suffer from training instabilities due to gradient mismatch from the Straight-Through Estimator (STE). This paper introduces BAMSProd, an optimization algorithm that stabilizes deep binary model training. By combining an adaptive gradient projection linked to quantization errors with the non-increasing second-moment strategy of AMSGrad, BAMSProd prevents error accumulation. Empirical results across vision and language tasks demonstrate 1.2x faster convergence and improved accuracy, achieving 70.91% Top-1 accuracy on ImageNet with MobileNetv2.
Introduction: The Challenge of Binary Optimization
Binary Neural Networks (BNNs) are a promising solution for deploying deep learning on resource-constrained devices, potentially offering up to 58x faster computation and 32x smaller model sizes by using 1-bit weights and activations. However, training these models is notoriously difficult.
The primary obstacle is the Straight-Through Estimator (STE). Since the binarization function (the sign function) has a derivative of zero almost everywhere, STE is used to approximate the derivative as 1 during backpropagation. This creates a fundamental mismatch between the estimated gradient and the actual loss surface. Standard adaptive optimizers like Adam exacerbate this issue; their exponential moving average (EMA) tracks past gradients, causing errors from STE to accumulate and often leading to suboptimal solutions or total failure to converge.
Methodology: BAMSProd
To address these issues, the authors propose BAMSProd (Binary Adaptive Moving Strategy with Projection). The algorithm integrates three key components:
- AMSGrad Foundation: It utilizes the AMSGrad strategy, enforcing a non-decreasing second-moment estimate ($v'_t = \max(v_t, v'_{t-1})$). This stabilizes the learning rate when gradients are sparse or noisy.
- Adaptive Range Projection: This novel mechanism projects the second-moment estimate into a dynamic range $[c_l, c_u]$. These bounds are functions of the quantization error—the difference between full-precision latent weights and their binary counterparts.
- Smoothing Transitions: By tying optimizer constraints to quantization error, BAMSProd smooths gradient transitions and prevents the erratic weight flips common in BNN training.
Experimental Results
The authors evaluated BAMSProd on several benchmarks, including ImageNet (MobileNetv2), CIFAR-10 (ResNet-34), Penn Treebank (LSTM), and PASCAL VOC (YOLOv2).
Performance and Convergence
On ImageNet, BAMSProd achieved 70.91% Top-1 accuracy, surpassing standard Adam (70.24%) and the specialized binary optimizer Bop (69.46%). In the highly non-convex task of object detection with YOLOv2, BAMSProd reached 73.9 mAP, whereas standard Adam failed significantly, reaching only 18.5 mAP.
BAMSProd also demonstrated 1.2x faster convergence on CIFAR-10 compared to existing methods, with a noticeably smoother loss curve and fewer oscillations. In language modeling tasks using LSTMs, it maintained stable perplexity, avoiding the wild fluctuations seen with other adaptive methods.
Conclusion
BAMSProd represents a significant step in making adaptive optimization robust for deep binary models. By treating quantization error as a control signal rather than a nuisance, it provides a principled way to stabilize training without the drawbacks of manual gradient clipping. Future work will look into extending these adaptive projections to multi-bit quantized models and bridging the theoretical gap for deep non-convex binary optimization.
Summary updated