Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, Luc Van Gool

TL;DR

This paper introduces a novel "soft-to-hard" annealing approach for training deep neural networks to learn compressible representations end-to-end. By smoothly transitioning from continuous approximations to discrete quantization during training, the method achieves state-of-the-art results in both image and neural network compression without relying on hand-crafted assumptions.

What Problem Does This Paper Address?

Modern deep neural networks (DNNs) are incredibly powerful but often come with a significant computational and memory footprint. This is due to their large number of parameters and the rich, often redundant, intermediate feature representations they generate. Compressing these models and their representations is crucial for deployment on resource-constrained devices and for efficient data storage.

Traditional compression methods for DNNs and images often involve separate, multi-step procedures. For instance, DNN compression might involve pruning weights, followed by quantization, and then retraining. Image compression often uses specialized codecs. The challenge is to develop a unified, end-to-end trainable framework that can learn representations that are inherently compressible, minimizing both distortion (loss of information) and rate (bits required to store the representation). A key hurdle in this endeavor is the non-differentiable nature of quantization and entropy estimation, which are fundamental to compression.

Methods

The core innovation of this paper is the soft-to-hard vector quantization approach. Instead of directly optimizing with discrete quantization and entropy, the authors introduce a continuous, differentiable relaxation.

Here's a breakdown of the key techniques:

  • Soft Assignments: The hard, discrete assignment of a vector to its nearest quantization center is replaced by a "soft" assignment using the softmax function. This creates a continuous probability distribution over the centers, allowing for gradient-based optimization. A "hardness" parameter ($\sigma$) controls how close this soft assignment is to a hard assignment.
  • Soft Entropy Estimation: Similarly, the discrete entropy of the symbol stream is approximated by a differentiable "soft entropy" based on these soft assignments.
  • Soft-to-Hard Annealing: The crucial element is the annealing process. The hardness parameter $\sigma$ is gradually increased from a small initial value to infinity during training. This allows the network to first learn using the smooth, differentiable soft approximations and then transition to the desired hard, discrete representations. This annealing strategy effectively bridges the gap between the continuous optimization landscape and the discrete nature of compression.
  • Vector Quantization: The paper extends previous work by employing vector quantization (VQ) instead of scalar quantization. This means that groups of features (vectors) are quantized together, allowing for the capture of local correlations and potentially leading to better compression.
  • No Distributional Assumptions: Unlike many prior methods that assume specific distributions for the data (e.g., Gaussian), this approach relies on histograms of assignment probabilities, making it more general.

Datasets

The paper evaluates its method on two main tasks:

  • Image Compression:
    • Training: A subset of 90,000 images from ImageNet [9] was used, downsampled by a factor of 0.7 and trained on 128x128 pixel crops.
  • Evaluation: The method was tested on four standard datasets: Kodak [2], B100 [31], Urban100 [14], and ImageNET100 (100 randomly selected images from ImageNet).
  • DNN Compression:
    • Evaluation: A 32-layer ResNet [13] model trained on CIFAR-10 [18] was used.

Key Insights and Ideas

  • Unified Framework: The paper elegantly unifies image and DNN compression under a single end-to-end learning framework, demonstrating the versatility of the soft-to-hard quantization approach.
  • Annealing as a Bridge: The soft-to-hard annealing is a key insight. It allows for gradient-based optimization of non-differentiable components by gradually making them "harder," effectively guiding the network towards a compressible discrete representation.
  • Vector Quantization Benefits: The application of vector quantization to learned compression is shown to be beneficial, outperforming scalar quantization in terms of compression rates for a given reconstruction error.
  • Generality through Histograms: By avoiding assumptions about data distributions and instead using learned histograms of assignments, the method becomes more robust and applicable to a wider range of data.

Key Results

  • Image Compression: The proposed Soft-to-Hard Vector Quantization Autoencoder (SHVQ) achieves competitive performance with state-of-the-art methods like BPG, JPEG 2000, and JPEG, especially at compression rates below 1 bit per pixel (bpp). For rates < 0.4 bpp, it outperforms JPEG and JPEG 2000 in MS-SSIM and is competitive with BPG. Visually, SHVQ-compressed images exhibit fewer artifacts than JPEG 2000.
  • DNN Compression: For a 32-layer ResNet on CIFAR-10, the method achieves a comparable test accuracy of 92.1% while compressing the DNN by a factor of 19.15 (with Huffman coding) and 20.15 (with arithmetic coding). This is achieved without the extensive manual steps (pruning, retraining, etc.) required by many prior DNN compression techniques.

Limitations and Future Work

  • Annealing Schedule: The authors acknowledge that choosing the right annealing schedule for $\sigma$ is crucial and can be sensitive. Annealing too slowly might lead to large weights, while annealing too fast can result in vanishing gradients.
  • Computational Cost: While the method is end-to-end trainable, the annealing process and the soft assignments can add computational overhead during training.
  • Future Directions: The paper suggests that the framework can be easily adapted for other tasks requiring compressible representations. The simplicity of the approach opens doors for exploring its application in various domains beyond image and DNN compression.

Summary updated

Abstract

We present a new approach to learn compressible representations in deep architectures with an end-to-end training strategy. Our method is based on a soft (continuous) relaxation of quantization and entropy, which we anneal to their discrete counterparts throughout training. We showcase this method for two challenging applications: Image compression and neural network compression. While these tasks have typically been approached with different methods, our soft-to-hard quantization approach gives results competitive with the state-of-the-art for both.
BibTeX
@inproceedings{agustsson2017soft,
  author = {Eirikur Agustsson and Fabian Mentzer and Michael Tschannen and Lukas Cavigelli and Radu Timofte and Luca Benini and Luc Van Gool},
  title = {{Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=p7tGiWdK6C},
}
Figure