Abstract
Deep Convolutional Neural Networks (CNNs) are notoriously memory-intensive, often exceeding the storage capacities of resource-constrained edge devices. While Binary Neural Networks (BNNs) provide a significant reduction in model size, they are theoretically capped at a 32x compression ratio. This paper introduces Stacking Low-dimensional Binary Filters (SLBF), a novel compression technique that breaks this ceiling. By approximating large convolution filters using a shared dictionary of small, 1-bit building blocks, SLBF achieves compression ratios as high as 261x on VGG-16 while maintaining accuracy within a few percentage points of full-precision models.
Introduction
The deployment of state-of-the-art CNNs like VGG-16, which requires over 500MB of storage, remains a challenge for mobile and IoT platforms. Existing solutions typically involve binarization, where 32-bit floating-point weights are replaced with 1-bit values (+1 or -1). However, because standard BNNs maintain the same number of parameters as the original model, they cannot exceed a ~32x compression ratio. To achieve higher efficiency, researchers often resort to pruning or low-rank approximations. SLBF seeks to combine the efficiency of binarization with the power of parameter sharing to push past these traditional limits.
Methodology
The core of the SLBF approach is the decomposition of full-sized filters into smaller, shared components.
- The Dictionary: For each layer, a small "dictionary" of $m$ low-dimensional binary filters is learned. These filters share the spatial dimensions of the target filter (e.g., 3x3) but have a significantly reduced depth ($s$) compared to the input channels ($c_{in}$).
- Stacking and Weight Sharing: A full-sized filter is reconstructed by selecting and "stacking" blocks from this shared dictionary. Since the dictionary is reused across all filters in a layer, the model only needs to store the small dictionary and an indicator matrix mapping blocks to their positions.
- Optimization: Training discrete indicator matrices is non-trivial. The authors demonstrate that stacking filters in the weight domain is mathematically equivalent to a "split-transform-merge" strategy in the feature map domain. They employ the Straight Through Estimator (STE) to backpropagate gradients through non-differentiable sign and argmax functions.
Experimental Setup
The SLBF method was evaluated on the CIFAR-10 and CIFAR-100 image classification benchmarks. The authors tested the approach using two prominent architectures: VGG-16 and ResNet-18. Performance was compared against standard full-precision models and state-of-the-art binarization methods such as XNOR-Net and Binary Weight Networks (BWN).
Key Insights
- Surpassing the 32x Limit: By forcing different filters to share the same binary "DNA" via dictionary blocks, SLBF reduces the total bit-count far more aggressively than standard BNNs.
- Computational Efficiency: The reformulation of stacking as a sum of intermediate feature maps ensures that the model remains efficient during both training and inference.
- Learned Scaling: Incorporating a simple learned scaling factor ($\alpha$) for each stacked block significantly boosts accuracy with negligible memory overhead.
- Batch Normalization: The study highlights that Batch Normalization with scaling and shifting is essential for the convergence of these highly compressed models.
Results
SLBF consistently outperforms existing binarization methods in compression-to-accuracy trade-offs. On VGG-16 (CIFAR-10), SLBF achieved a 261.4x compression ratio with 89.24% accuracy, far exceeding the ~31x compression of standard BNNs. At a more moderate 103.2x compression ratio, the model maintained 91.30% accuracy, representing less than a 2% drop from the full-precision baseline (93.25%). Similar gains were observed on ResNet-18 (CIFAR-100), where SLBF reached 95.1x compression with 74.19% accuracy.
Conclusion and Future Work
SLBF demonstrates that combining binarization with parameter sharing can yield extreme compression without catastrophic accuracy loss. While this work focuses on weight compression, future research could incorporate binary activations to further accelerate inference via bitwise operations. Additionally, evaluating the method on larger datasets like ImageNet and exploring optimization alternatives to the Straight Through Estimator remain important areas for future development.
Summary updated