Hybrid Binary Networks: Optimizing for Accuracy, Efficiency and Memory

Ameya Prabhu, Vishal Batchu, Rohit Gajawada, Sri Aurobindo Munagala, Anoop M. Namboodiri

TL;DR

Hybrid Binary Networks (HBNs) offer a "best of both worlds" approach to neural network compression by selectively binarizing activations only in specific layers. This data-driven strategy recovers significant accuracy lost in fully binarized models while maintaining massive gains in memory and computational efficiency, making it ideal for deploying AI on resource-constrained edge devices.

What Problem Does This Paper Address?

Deep learning models are notoriously "heavy," requiring massive amounts of memory and computational power (FLOPs). This makes them difficult to run on mobile phones or embedded systems. Binarization—reducing weights and activations to just 1 bit (0 or 1)—is the most extreme form of compression, offering up to 58x speedups and 32x memory savings.

However, there is a catch: binarizing everything usually leads to a "performance cliff" where accuracy drops too much for practical use. The authors tackle the "where to binarize" problem. They argue that not all layers are created equal; some are highly sensitive to binarization noise, while others are robust. The challenge is finding an automated, efficient way to decide which layers should stay at higher precision to preserve accuracy without sacrificing the benefits of binarization.

Methods

The authors introduce a framework for creating Hybrid Binary Networks. Their approach involves three main steps:

  1. The Metric: They propose a joint metric ($M$) that balances binarization error ($E$) with computational cost ($1/NF$). This metric helps identify layers where the "cost" of binarization (in terms of accuracy loss) is too high relative to the "gain" (in terms of speed).
  2. The Partitioning Algorithm: Using a pre-trained fully binarized model, they calculate this metric for every layer. They then use K-means clustering to group layers. Layers in the "highest error" cluster are kept as WeightBinConv (binary weights, but full-precision inputs), while the rest are converted to FullBinConv (both weights and inputs are binary).
  3. Hybridization Ratio ($R$): This hyperparameter allows engineers to tune the tradeoff. A low $R$ prioritizes speed (more binary layers), while a higher $R$ prioritizes accuracy.

Crucially, unlike many previous methods that keep the first and last layers in full precision by default, this algorithm uses data to decide, often finding that binarizing the weights of the last layer is perfectly fine if the inputs are handled correctly.

Datasets

The researchers evaluated their approach on three major benchmarks:

  • ImageNet: The standard large-scale dataset for image classification.
  • TU-Berlin: A popular dataset of human-drawn sketches.
  • Sketchy: A recent, large-scale dataset of hand-drawn sketches used to cross-validate the results from TU-Berlin.

The focus on sketch datasets is a unique contribution, as sketches are inherently sparse and "binary-ish," making them an interesting use case for binarized networks.

Key Insights and Ideas

  • Layer-wise Independence: The authors argue that binarization noise doesn't propagate through the network as much as previously feared. This means we can make decisions about one layer's precision without worrying too much about how it affects the next, simplifying the optimization process.
  • The Last Layer is Compressible: Traditional wisdom says you must keep the last layer in full precision. This paper shows that you can binarize the weights of the last layer to save massive amounts of memory, provided you keep the inputs to that layer at full precision.
  • Natural Fit for Sketches: Binarization is particularly effective for sketch recognition. Because sketches lack the complex textures and colors of real-world photos, the "information loss" from binarization is much lower.
  • Synergy with Compact Architectures: The method isn't just for "fat" models like AlexNet. It can be applied to already-compressed models like SqueezeNet, achieving an overall compression of 256x with only a 6% drop in accuracy.

Key Results

  • Accuracy Gains: On ImageNet, HBNs achieved a 3-4% improvement in accuracy over fully binarized AlexNet and ResNet-18 models. On sketch datasets, the improvement was even more dramatic, exceeding 8%.
  • Extreme Compression: For the Sketch-A-Net model, the hybrid approach achieved a staggering 233x memory compression compared to the full-precision version while matching state-of-the-art accuracy.
  • Efficiency: The hybrid models maintained significant speedups, typically requiring only a marginal increase in FLOPs (e.g., 1.4x to 2.7x) compared to the "base" binarized models, which is still vastly faster than full-precision inference.

Limitations and Future Work

  • Training from Scratch: Currently, once the hybrid architecture is decided, the model must be trained from scratch. A method to "fine-tune" into a hybrid state might be more efficient.
  • Hyperparameter Tuning: The hybridization ratio ($R$) and the tradeoff ratio ($\gamma$) still require some manual tuning or grid search to find the optimal balance for a specific hardware target.
  • Hardware Implementation: While the paper calculates theoretical speedups based on FLOPs, the authors note that actual real-world speedups depend heavily on specific hardware kernels for bitwise operations, which is a suggested area for future exploration.

Summary updated

Abstract

Binarization is an extreme network compression approach that provides large computational speedups along with energy and memory savings, albeit at significant accuracy costs. We investigate the question of where to binarize inputs at layer-level granularity and show that selectively binarizing the inputs to specific layers in the network could lead to significant improvements in accuracy while preserving most of the advantages of binarization. We analyze the binarization tradeoff using a metric that jointly models the input binarization-error and computational cost and introduce an efficient algorithm to select layers whose inputs are to be binarized. Practical guidelines based on insights obtained from applying the algorithm to a variety of models are discussed. Experiments on Imagenet dataset using AlexNet and ResNet-18 models show 3-4% improvements in accuracy over fully binarized networks with minimal impact on compression and computational speed. The improvements are even more substantial on sketch datasets like TU-Berlin, where we match state-of-the-art accuracy as well, getting over 8% increase in accuracies. We further show that our approach can be applied in tandem with other forms of compression that deal with individual layers or overall model compression (e.g., SqueezeNets). Unlike previous quantization approaches, we are able to binarize the weights in the last layers of a network, which often have a large number of parameters, resulting in significant improvement in accuracy over fully binarized models.
BibTeX
@inproceedings{prabhu2018hybrid,
  author = {Ameya Prabhu and Vishal Batchu and Rohit Gajawada and Sri Aurobindo Munagala and Anoop M. Namboodiri},
  title = {{Hybrid Binary Networks: Optimizing for Accuracy, Efficiency and Memory}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=bHxIKqd-30},
}

Loading related content...

Figure