Partial Binarization of Neural Networks for Budget-Aware Efficient Learning

Udbhav Bamba, Neeraj Anand, Saksham Aggarwal, Dilip K. Prasad, Deepak K. GuptaWACV 2024

TL;DR

This paper introduces Budgeted Binary Neural Networks (B2NNs) and a novel strategy called MixBin for efficiently compressing deep neural networks. MixBin intelligently selects which layers to binarize and which to keep in full precision, achieving significant reductions in computational cost (FLOPs) with minimal performance degradation, outperforming existing methods on challenging datasets like ImageNet.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (CNNs) are incredibly powerful for computer vision tasks, but their computational demands are immense. This leads to high energy consumption and slow inference times, making them impractical for deployment on resource-constrained devices. While binarization (reducing weights and activations to 1-bit) offers substantial compression, it often results in a drastic drop in accuracy. Existing partial binarization methods lack a systematic approach to balance compression and performance, making it difficult to find the optimal mix of binary and full-precision components for a given computational budget.

Methods

The core contribution is the MixBin strategy, designed to create Budgeted Binary Neural Networks (B2NNs). Unlike random or iterative layer selection, MixBin offers a controlled approach to partial binarization. The method works by:

  • Identifying Layer Sensitivity: MixBin analyzes how sensitive a network's performance is to binarizing individual layers. This is done by considering the drop in performance when a single layer is binarized while the rest remain in full precision.
  • Binarization Coefficient Calculation: Two variants of MixBin are proposed:
  • MixBinLoss: Calculates a coefficient based on the difference in loss when a layer is binarized versus when it's not. A smaller difference suggests the layer is a good candidate for binarization.
  • MixBinGrad: Extends MixBinLoss by also incorporating the L1 norm of the gradient of the loss with respect to the layer's parameters. This aims to identify layers that, once binarized, have less influence on the model's performance.
  • Budgeted Selection: The calculated coefficients are used to sort layers, and a subset is chosen for binarization to meet a predefined FLOPs budget. This allows for explicit control over the compression level.

The paper also contrasts MixBin with several trivial baselines: random selection, front/rear layer binarization, iterative greedy selection, and pseudo-random selection.

Datasets

The proposed methods were evaluated on the following datasets:

  • CIFAR-100: Used with the cResNet-20 architecture.
  • TinyImageNet: Used with MobileNet and ResNet-18 architectures.
  • ImageNet-1K: Used with the ResNet-18 architecture for large-scale classification.
  • GOT-10K: Used for object tracking with the ResNet-18 model.

Key Insights and Ideas

  • Layer Importance Varies Drastically: Binarizing different parts of a CNN has a highly uneven impact on performance. The paper's experiments show that binarizing later layers can be detrimental, while keeping them in full precision often boosts performance.
  • Systematic Approach is Crucial: Random or simple sequential selection of layers for binarization is insufficient. A more intelligent strategy like MixBin is needed to effectively balance compression and accuracy.
  • MixBin Offers Fine-Grained Control: The ability to explicitly choose the fraction of the network to be binarized (budgeted binarization) is a significant advantage for deploying models under strict computational constraints.
  • Transferability of B2NNs: Models compressed using MixBin on one dataset (e.g., ImageNet) can be transferred to other datasets (e.g., TinyImageNet, CIFAR-100) and tasks (object tracking) with competitive performance, suggesting good generalization.

Key Results

  • ImageNet-1K Performance: B2NNs created using MixBin outperformed random and iterative searches, as well as state-of-the-art layer selection methods, by up to 3% on ImageNet-1K.
  • Extreme Compression: At a very low FLOPs budget (15%), B2NNs achieved approximately 23% better performance than structured pruning baselines.
  • Object Tracking: MixBin-based B2NNs showed up to a 12.4% relative improvement in object tracking performance compared to other baselines.
  • Generalization: Transferred B2NN models showed performance comparable to models trained directly on the target dataset, even at low budgets. For instance, at a 20% FLOPs budget, transferred models sometimes even slightly outperformed base-trained models on classification tasks.

Limitations and Future Work

  • Layer-Level Optimization: The current MixBin strategy focuses on optimizing binarization at the layer level. The authors suggest future work could explore more granular optimization at the channel level.
  • Transformer Architectures: Extending the layer selection method to transformer-based architectures is proposed as an interesting avenue for future research.
  • Hardware Limitations: The paper acknowledges that current hardware is not fully optimized for binary neural networks, which limits the complete exploitation of B2NNs. However, they are optimistic that rapid hardware advancements will address this.

Summary updated

BibTeX
@inproceedings{bamba2024partial,
  author = {Udbhav Bamba and Neeraj Anand and Saksham Aggarwal and Dilip K. Prasad and Deepak K. Gupta},
  title = {{Partial Binarization of Neural Networks for Budget-Aware Efficient Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure