Fast and Accurate Binary Neural Networks Based on Depth-Width Reshaping

Ping Xue, Yang Lu, Jingfei Chang, Xing Wei, Zhen WeiAAAI 2023 · DOI

Abstract

Binary Neural Networks (BNNs) represent the extreme limit of model compression, utilizing 1-bit weights and activations to minimize storage and maximize computational speed. However, BNNs typically suffer from a significant accuracy gap compared to full-precision counterparts. This paper identifies that standard "deep and narrow" architectures, while optimal for high-precision math, are suboptimal for 1-bit networks due to quantization error accumulation. We propose Depth-Width Reshaping (DWR), a framework that redesigns BNN backbones to be shallower and wider. Our approach achieves state-of-the-art accuracy on ImageNet while simultaneously improving inference speed by reducing activation overhead.

Introduction

Binary Neural Networks are often considered the "holy grail" of efficient deep learning. By quantizing parameters to single bits, they enable massive speedups on specialized hardware. Despite these advantages, the "accuracy gap" remains a hurdle. Most research attempts to bridge this gap through improved training heuristics or loss functions, yet almost all rely on standard backbones like ResNet-18 or VGG.

This paper argues that the fundamental shape of these backbones is flawed for BNNs. In full-precision models, depth is a primary driver of representational power. In BNNs, however, every layer introduces quantization noise and gradient approximation errors (via the Straight-Through Estimator). As a network grows deeper, these errors accumulate, eventually negating the benefits of the additional layers.

Methodology: Depth-Width Reshaping (DWR)

We propose a two-step approach to redesigning BNN architectures to mitigate error accumulation and hardware bottlenecks.

Conceptual overview of the transition from deep/narrow to shallow/wide architectures followed by pruning.

1. Naive Reshaping

We utilize a mathematical scaling rule to maintain a constant computational budget (Operations) while altering the network's aspect ratio. Specifically, we decrease the depth ($d$) by a factor of $s$ and increase the width ($w$) by $\sqrt{s}$. This transformation results in a shallower network that is more robust to the "error accumulation tax" inherent in binary optimization.

2. Reshaping with Pruning

To refine the coarse adjustments of naive reshaping, we incorporate clustering-based pruning. This serves as an automated architecture search, allowing the model to fine-tune layer widths to find the most efficient configuration that fits within the original computational budget.

Detailed architectural implementation of DWR on a residual network, showing channel scaling and stage reduction.

Key Insights

  • Width vs. Depth: Increasing width provides representational capacity without adding new stages of quantization error. This makes width a "safer" scaling dimension for binary models compared to depth.
  • Activation Efficiency: While many optimization efforts focus on OPs (Operations), memory bandwidth is often the real-world bottleneck. By making the network shallower and wider, DWR reduces total activations by a factor of $O(\sqrt{s})$, leading to tangible speedups on modern hardware.

Experimental Results

The authors evaluated DWR on CIFAR-10 and ImageNet across multiple architectures (VGG-11, ResNet-18, ResNet-20) and BNN techniques (IR-Net, ReActNet).

  • State-of-the-Art Accuracy: On ImageNet, a DWR-optimized ResNet-18 achieved a Top-1 accuracy of 67.2%, significantly outperforming the ReActNet baseline of 65.5%.
  • Inference Speed: DWR models were approximately 12.5% faster to run than original backbones, despite their higher accuracy, due to the reduction in activation volume.
  • Consistency: The performance gains were consistent across different binarization methods, proving that backbone shape is a fundamental, and previously overlooked, bottleneck in BNN design.

Conclusion

Depth-Width Reshaping demonstrates that BNNs require a fundamental departure from full-precision architectural conventions. By trading depth for width, researchers can minimize quantization error and activation overhead. While currently focused on CNNs, future work will investigate how these trade-offs apply to emerging architectures like binarized Vision Transformers.

Summary updated

BibTeX
@inproceedings{xue2023fast,
  author = {Ping Xue and Yang Lu and Jingfei Chang and Xing Wei and Zhen Wei},
  title = {{Fast and Accurate Binary Neural Networks Based on Depth-Width Reshaping}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26268},
}
Figure