Enhancing Post-Training Quantization Calibration Through Contrastive Learning

Yuzhang Shang, Gaowen Liu, Ramana Rao Kompella, Yan Yan

Introduction

Post-Training Quantization (PTQ) is a critical technique for deploying deep neural networks on resource-constrained edge devices. By converting 32-bit floating-point (FP) weights and activations into low-precision integers (e.g., 4-bit or 2-bit), PTQ significantly reduces memory footprint and computational latency. Unlike Quantization-Aware Training (QAT), PTQ is highly efficient as it requires no full retraining and only a small set of unlabeled calibration data.

However, as bit-widths drop to ultra-low levels, standard PTQ methods often suffer from severe accuracy degradation. Traditional calibration techniques typically rely on minimizing simple distance metrics, such as Mean Squared Error (MSE) or Cosine Similarity, between FP and quantized activations. The authors of this CVPR paper argue that these metrics are insufficient because they focus on numerical alignment rather than the preservation of underlying semantic information.

The CL-Calib Framework

To address these limitations, the researchers introduce CL-Calib (Contrastive Learning Calibration), a "plug-and-play" module designed to maximize information retention during the quantization process.

Conceptual overview of CL-Calib, showing how activations are pulled together or pushed apart in a contrastive space to preserve information during quantization.

The core idea is to treat quantization calibration as a self-supervised contrastive learning task. Instead of just matching values, CL-Calib ensures that the quantized model's internal representations remain informative relative to the original FP model.

Technical Implementation

The framework operates by defining positive and negative pairs of activations:

  1. Positive Pairs: The quantized activation and the corresponding FP activation derived from the same input image. These are "pulled" together in the feature space.
  2. Negative Pairs: Activations derived from different images, which are "pushed" apart to maintain discriminative power.

A significant challenge in applying contrastive learning to PTQ is the lack of data; contrastive methods usually require large batches and extensive training. CL-Calib overcomes this by using the remaining layers of the pre-trained FP model as a "critic function."

Detailed architecture of the CL-Calib pipeline, illustrating the use of the frozen full-precision network as a critic to guide the calibration of quantized layers.

As shown in the diagram above, the frozen FP network acts as a fixed encoder that projects activations into a space where their semantic similarity can be judged. This allows the model to optimize quantization parameters (scaling factors and zero points) based on how well they preserve the features necessary for the final prediction.

Key Insights

  • Mutual Information Maximization: The authors provide a theoretical proof that minimizing the contrastive loss (InfoNCE) is equivalent to maximizing the lower bound of Mutual Information (MI) between the FP and quantized activations. This shifts the objective from "matching numbers" to "preserving signal."
  • Stability and Regularization: By using the frozen FP network as a critic, the method avoids the common "collapse" issue in contrastive learning. Furthermore, CL-Calib acts as a regularizer, preventing the quantization parameters from overfitting to the small calibration dataset (typically only 128 images).

Experimental Results

The primary evaluation on ImageNet demonstrates that CL-Calib consistently enhances state-of-the-art PTQ methods like QDrop and PD-Quant. The improvements are most pronounced in challenging 2-bit (W2A2) configurations:

  • MobileNetV2: Accuracy improved by +4.20%.
  • MNasNet: Accuracy increased by +5.30%.
  • ResNet-18: Observed a +3.21% gain.

The method also showed strong generalization across different architectures, including Vision Transformers (ViT) and tasks like object detection.

Conclusion

CL-Calib represents a significant step forward for ultra-low-bit quantization. By leveraging contrastive learning to preserve mutual information, it allows models to maintain high accuracy even when compressed to 2 bits. While it introduces some computational overhead during the calibration phase and requires tuning of loss-weighting hyper-parameters, its ability to recover accuracy without retraining makes it a powerful tool for edge AI deployment.

Summary updated

Abstract

Post-training quantization (PTQ) converts a pre-trained full-precision (FP) model into a quantized model in a training-free manner. Determining suitable quantization parameters, such as scaling factors and weight rounding, is the primary strategy for mitigating the impact of quantization noise (calibration) and restoring the performance of the quantized models. However, the existing activation calibration methods have never considered information degradation between pre- (FP) and post-quantized activations. In this study, we introduce a well-defined distributional metric from information theory, mutual information, into PTQ calibration. We aim to calibrate the quantized activations by maximizing the mutual information between the pre- and post-quantized activations. To realize this goal, we establish a contrastive learning (CL) framework for the PTQ calibration, where the quantization parameters are optimized through a self-supervised proxy task. Specifically, by leveraging CL during the PTQ process, we can benefit from pulling the positive pairs of quantized and FP activations collected from the same input samples, while pushing negative pairs from different samples. Thanks to the ingeniously designed critic function, we avoid the unwanted but often-encountered collision solution in CL, especially in calibration scenarios where the amount of calibration data is limited. Additionally, we provide a theoretical guarantee that minimizing our designed loss is equivalent to maximizing the desired mutual information. Consequently, the quantized activations retain more information, which ultimately enhances the performance of the quantized network. Experimental results show that our method can effectively serve as an add-on module to existing SoTA PTQ methods.
BibTeX
@inproceedings{shang2024enhancing,
  author = {Yuzhang Shang and Gaowen Liu and Ramana Rao Kompella and Yan Yan},
  title = {{Enhancing Post-Training Quantization Calibration Through Contrastive Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01507},
}
Figure