TL;DR
This paper introduces GradFreeBits, a novel method for optimizing mixed-precision neural networks by decoupling weight optimization (gradient-based) from bit allocation optimization (gradient-free using CMA-ES). This approach effectively balances accuracy and computational cost, achieving state-of-the-art results across various tasks with reduced training complexity.
What Problem Does This Paper Address?
Deploying deep neural networks (DNNs) on resource-constrained edge devices is a significant challenge due to their high computational and memory demands. Quantization, particularly mixed-precision quantization (using different bit widths for different layers), offers a promising solution by reducing these requirements. However, optimizing the bit allocation for each layer is a complex, discrete optimization problem that is difficult to solve with standard gradient-based methods. Existing approaches often struggle to effectively capture the interdependencies between layer bit allocations and hardware constraints, leading to suboptimal trade-offs between accuracy and efficiency.
Methods
The core of GradFreeBits is an iterative alternating retraining scheme. It combines two distinct optimization processes:
- Gradient-based optimization for weights: Standard quantization-aware training is used to update the network's weights.
- Gradient-free optimization for bit allocation: The Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a powerful gradient-free optimization algorithm, is employed to find the optimal bit allocation for each layer. CMA-ES is chosen for its ability to handle discontinuous and ill-conditioned optimization landscapes, which are characteristic of bit allocation problems.
The bit allocation is optimized in the logarithmic space of the bit widths to create a smoother search space. The objective function for CMA-ES is a combination of the network's performance (e.g., accuracy) and computational constraints (e.g., model size), using a penalty method to incorporate these constraints.
To further improve training stability and performance, GradFreeBits also introduces:
- Bit-dependent clipping parameters: Clipping parameters are no longer fixed but are parameterized to adapt to varying bit allocations, preventing performance degradation during optimization.
- Moving Super-Batches: This technique reduces subsampling noise during objective function evaluation by using a queue-like replacement of minibatches, leading to more stable CMA-ES updates.
Datasets
The paper evaluates GradFreeBits on several standard datasets:
- Image Classification: CIFAR-10, CIFAR-100, and ImageNet.
- Semantic Segmentation: Cityscapes.
- Graph Node Classification: Cora, Pubmed, and Citeseer.
Key Insights and Ideas
- Decoupling optimization is key: Separating the gradient-based weight optimization from the gradient-free bit allocation optimization is crucial for effectively tackling the discrete nature of bit allocation.
- CMA-ES is a strong candidate for bit allocation: Its ability to handle complex, non-differentiable search spaces makes it well-suited for finding optimal bit configurations.
- Logarithmic bit allocation simplifies search: Optimizing the logarithm of bit widths creates a more continuous and manageable search space for CMA-ES.
- Bit-dependent clipping parameters are essential: Adapting clipping parameters to the current bit allocation during training prevents performance drops and improves the robustness of the optimization process.
- Variance reduction techniques (moving super-batches) improve CMA-ES stability: This leads to more reliable convergence and better final results.
- Lower training cost is achievable: The combination of efficient optimization algorithms and techniques like moving super-batches significantly reduces the overall training time compared to existing methods.
Key Results
GradFreeBits achieves state-of-the-art or comparable performance across various tasks:
- Image Classification: Outperforms existing methods on CIFAR-10/100 and ImageNet, often achieving better accuracy with smaller model sizes. For example, on ImageNet, the 3W/3A ResNet18 model achieves a smaller model size and higher Top1 accuracy compared to previous methods.
- Semantic Segmentation: Achieves strictly superior model size and mIoU on Cityscapes. The 8-bit DeepLabV3(ResNet50) model shows a significant improvement in mIoU with a considerably smaller model size.
- Graph Node Classification: Demonstrates significantly higher accuracy for comparable compression rates on semi-supervised node classification benchmarks.
- Computational Cost: Significantly reduces training cost, requiring fewer epochs compared to other methods (e.g., 57 effective epochs for ImageNet vs. 100-200 for others).
Limitations and Future Work
- Limitations: The paper acknowledges that the derivation of optimal clipping parameters is based on approximations (e.g., Laplace distribution assumption) and that the linear approximation of clipping parameters might benefit from richer parameterization. While CMA-ES is efficient, it can still be computationally expensive for very high-dimensional problems (though the authors argue it's efficient for the bit allocation problem).
- Future Work: The authors suggest extending the framework to incorporate additional constraints, such as hardware simulator measurements. They also propose exploring similar iterative retraining approaches for other hyperparameter optimization tasks, like per-layer pruning rates, to achieve similar trade-offs between accuracy and computational cost.
Summary updated