Soft-to-Hard Vector Quantization for End-to-End Learned Compression of Images and Neural Networks

Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, Luc Van Gool

Abstract

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.

Introduction

Deep learning models face two major compression challenges: reducing model size for deployment on resource-constrained devices and developing data codecs that outperform traditional standards like JPEG. The fundamental hurdle in both cases is quantization. Mapping continuous values to discrete symbols is a non-differentiable operation, which prevents the use of standard backpropagation. Additionally, optimizing for entropy—the average bits per symbol—requires a differentiable way to estimate and minimize symbol distributions.

Methods: Soft-to-Hard Vector Quantization

The authors propose a unified framework that treats image and model compression as mathematically equivalent problems. The core of the approach is a Soft-to-Hard Vector Quantization strategy.

Unified framework for model (top) and image (bottom) compression using soft-to-hard quantization.

Continuous Relaxation and Annealing

Instead of hard quantization, the network uses "soft" assignments where a value is represented as a weighted average of learnable quantization centers using a softmax-based function. A "hardness" parameter ($\sigma$) is introduced and gradually increased during training. This deterministic annealing process allows the network to initially explore the space with smooth gradients and eventually converge to a discrete, "hard" representation. This approach also utilizes Vector Quantization (VQ), which captures local correlations more effectively than scalar methods.

Differentiable Entropy Estimation

To minimize the final bit rate, the model must minimize entropy. The authors use "soft histograms" by aggregating soft assignments across a batch. This provides a differentiable estimate of the symbol distribution, enabling the joint optimization of distortion ($D$) and rate ($R$) via the loss function $L = D + \beta R$ without assuming a fixed mathematical distribution for the data.

Experimental Evaluation

The framework was evaluated on two distinct tasks: image compression using a subset of ImageNet (evaluated on benchmarks like B100) and model compression of a 32-layer ResNet on CIFAR-10.

Key Results

In image compression, the Soft-to-Hard Autoencoder (SHA) significantly outperforms legacy standards like JPEG and JPEG 2000 at low bitrates (below 0.4 bpp). As shown in the rate-distortion curves, the performance is competitive with BPG, a modern traditional codec.

Rate-distortion performance (MS-SSIM) on the B100 dataset compared to traditional codecs.

For model compression, the method achieved a 20x compression factor for ResNet-32, reducing weights to approximately 1.6 bits each while maintaining a test accuracy of 92.1% (a negligible drop from the uncompressed 92.6%). Unlike competing techniques that require multi-stage pipelines, this method is end-to-end and significantly simpler to implement.

Conclusion

The soft-to-hard quantization framework provides a simple, end-to-end solution for learning compressible representations. By unifying model and data compression, it demonstrates that the same underlying principles can achieve state-of-the-art results across different domains. While the annealing schedule requires careful tuning to avoid vanishing gradients, the framework offers a robust path forward for learned compression in complex architectures.

Summary updated

Abstract

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.
BibTeX
@inproceedings{agustsson2017soft,
  author = {Eirikur Agustsson and Fabian Mentzer and Michael Tschannen and Lukas Cavigelli and Radu Timofte and Luca Benini and Luc Van Gool},
  title = {{Soft-to-Hard Vector Quantization for End-to-End Learned Compression of Images and Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=p7tGiWdK6C},
}
Figure