Abstract
We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.
Introduction
Deep learning models face two major compression challenges: reducing model size for deployment on resource-constrained devices and developing data codecs that outperform traditional standards like JPEG. The fundamental hurdle in both cases is quantization. Mapping continuous values to discrete symbols is a non-differentiable operation, which prevents the use of standard backpropagation. Additionally, optimizing for entropy—the average bits per symbol—requires a differentiable way to estimate and minimize symbol distributions.
Methods: Soft-to-Hard Vector Quantization
The authors propose a unified framework that treats image and model compression as mathematically equivalent problems. The core of the approach is a Soft-to-Hard Vector Quantization strategy.
Continuous Relaxation and Annealing
Instead of hard quantization, the network uses "soft" assignments where a value is represented as a weighted average of learnable quantization centers using a softmax-based function. A "hardness" parameter ($\sigma$) is introduced and gradually increased during training. This deterministic annealing process allows the network to initially explore the space with smooth gradients and eventually converge to a discrete, "hard" representation. This approach also utilizes Vector Quantization (VQ), which captures local correlations more effectively than scalar methods.
Differentiable Entropy Estimation
To minimize the final bit rate, the model must minimize entropy. The authors use "soft histograms" by aggregating soft assignments across a batch. This provides a differentiable estimate of the symbol distribution, enabling the joint optimization of distortion ($D$) and rate ($R$) via the loss function $L = D + \beta R$ without assuming a fixed mathematical distribution for the data.
Experimental Evaluation
The framework was evaluated on two distinct tasks: image compression using a subset of ImageNet (evaluated on benchmarks like B100) and model compression of a 32-layer ResNet on CIFAR-10.
Key Results
In image compression, the Soft-to-Hard Autoencoder (SHA) significantly outperforms legacy standards like JPEG and JPEG 2000 at low bitrates (below 0.4 bpp). As shown in the rate-distortion curves, the performance is competitive with BPG, a modern traditional codec.
For model compression, the method achieved a 20x compression factor for ResNet-32, reducing weights to approximately 1.6 bits each while maintaining a test accuracy of 92.1% (a negligible drop from the uncompressed 92.6%). Unlike competing techniques that require multi-stage pipelines, this method is end-to-end and significantly simpler to implement.
Conclusion
The soft-to-hard quantization framework provides a simple, end-to-end solution for learning compressible representations. By unifying model and data compression, it demonstrates that the same underlying principles can achieve state-of-the-art results across different domains. While the annealing schedule requires careful tuning to avoid vanishing gradients, the framework offers a robust path forward for learned compression in complex architectures.
Summary updated