Flexible Network Binarization with Layer-wise Priority

Hongteng Xu 许洪腾ICIP 2017 · DOI

TL;DR

This paper introduces a novel "Layer-wise Priority" (LWB) method for binarizing neural networks. By strategically binarizing layers based on their impact on performance and compression, LWB achieves significant memory savings and speedups with minimal accuracy loss, outperforming existing methods like Binary-Weight-Networks (BWN).

What Problem Does This Paper Address?

Deep neural networks, while powerful, are computationally expensive and memory-intensive, limiting their deployment on resource-constrained devices like mobile phones and FPGAs. Network binarization offers a solution by reducing model size and speeding up computation by converting network weights from floating-point numbers to binary values (+1 or -1). However, existing binarization methods often binarize all layers simultaneously, leading to a significant drop in performance due to aggregated approximation errors. Furthermore, these methods lack flexibility in balancing the trade-off between compression ratio and performance degradation.

Methods

The core of this paper is the proposed Layer-wise Binarization (LWB) method. It addresses the limitations of simultaneous binarization by introducing a layer-wise priority concept. The key ideas are:

  • Layer-wise Priority: The paper observes that binarizing different layers has a varied impact on performance and compression. Deeper layers tend to offer higher compression with less performance loss. LWB leverages this by binarizing layers in inverse order of their depth.
  • Iterative Binarization and Fine-tuning: In each training step, LWB selects a specific layer (or a batch of layers) with the highest priority, minimizes the discrepancy between its real-valued weights and binary approximations, and then fine-tunes the entire network.
  • Flexible Trade-off: LWB allows for flexibility in deciding when to stop the binarization process. A criterion (e.g., a predefined threshold for miss rate increase) can be used to halt binarization, enabling a controlled trade-off between performance and compression.
  • Batch Processing for Acceleration: To speed up the process, LWB proposes binarizing a batch of high-priority (deep) layers in the initial steps, rather than one by one. This allows for more accelerated binary forward/backward propagations early on.

Datasets

  • Caltech: Used for training and evaluation of pedestrian detection.
  • INRIA: Used for training and evaluation of pedestrian detection.

Key Insights and Ideas

  • Layer Importance is Not Uniform: The impact of binarizing a layer is highly dependent on its position in the network. This is a crucial insight that existing methods overlooked.
  • Deeper Layers are "Safer" to Binarize: Binarizing deeper layers generally leads to higher compression with less accuracy degradation, making them ideal candidates for early binarization.
  • Iterative Refinement is Key: Instead of a one-shot binarization, LWB's iterative approach of binarizing a layer and then fine-tuning the whole network allows for better convergence and performance recovery.
  • Flexibility is Achievable: The proposed method offers a practical way to control the binarization process, allowing users to define acceptable performance trade-offs.

Key Results

  • Significant Compression: The LWB method achieved a model size reduction from 268.2 MB to as low as 8.7 MB for the YOLOv2 network.
  • Competitive Accuracy: For pedestrian detection on Caltech and INRIA datasets, LWB achieved miss rates of 26.9% and 14% respectively, outperforming many state-of-the-art methods and showing comparable or better results than BWN at similar compression levels.
  • Speed Improvement: The binarized YOLOv2 network achieved a speed of 66 FPS, a significant improvement over other detection algorithms like MSCNN (8 FPS).
  • Faster Convergence: LWB converges more quickly than BWN, requiring fewer training epochs to reach optimal performance.

Limitations and Future Work

  • Convergence Proof: The authors acknowledge that their method is not strictly alternating optimization or block coordinate descent, making a formal convergence proof challenging. However, they empirically demonstrate good convergence.
  • Future Work: The authors aim to develop more sophisticated metrics for layer priority and more effective criteria for evaluating and controlling the binarization process to further optimize the trade-off between compression and accuracy. They also plan to explore applying LWB to other neural network architectures.

Summary updated

BibTeX
@inproceedings{2017flexible,
  author = {Hongteng Xu 许洪腾},
  title = {{Flexible Network Binarization with Layer-wise Priority}},
  booktitle = {ICIP},
  year = {2017},
  doi = {10.1109/icip.2018.8451576},
}
Figure