Abstract
Post-training quantization methods use a set of calibration data to compute quantization ranges for network parameters and activations. The calibration data usually comes from the training dataset which could be inaccessible due to sensitivity of the data. In this work, we want to study such a problem: can we use out-of-domain data to calibrate the trained networks without knowledge of the original dataset? Specifically, we go beyond the domain of natural images to include drastically different domains such as X-ray images, satellite images and ultrasound images. We find cross-domain calibration leads to surprisingly stable performance of quantized models on 10 tasks in different image domains with 13 different calibration datasets. We also find that the performance of quantized models is correlated with the similarity of the Gram matrices between the source and calibration domains, which can be used as a criterion to choose calibration set for better performance. We believe our research opens the door to borrow cross-domain knowledge for network quantization and compression.
Introduction
To deploy deep neural networks on edge devices, quantization is used to convert high-precision 32-bit floating-point weights into low-precision integers (e.g., 8-bit). Post-Training Quantization (PQ) is a popular approach because it does not require retraining the entire model. However, PQ typically requires a small "calibration dataset" to determine the dynamic ranges of activations.
Conventionally, this data is sampled from the original training set. However, in many real-world scenarios, the original training data is inaccessible due to privacy (e.g., medical records), security, or proprietary restrictions. While "data-free" methods exist to synthesize calibration data from scratch, they are often computationally expensive and difficult to tune. This study investigates a simpler alternative: Cross-Domain Calibration, using unrelated real-world images to calibrate models.
Methodology
The proposed pipeline for cross-domain calibration involves three key components:
1. Naive Cross-Domain Calibration
This involves simply using images from a completely different domain (e.g., using natural images from Open Images) to calibrate a model trained on a specialized domain (e.g., X-rays).
2. BatchNorm (BN) Adjustment
The core technical contribution is the adjustment of BatchNorm layers. The authors argue that BN layers act as "domain anchors," storing statistics specific to the training data. To bridge the domain gap, they:
- Reset the running means and variances of all BN layers in the pre-trained float model.
- Pass the out-of-domain calibration data through the model to accumulate new BN statistics.
- Fold these adjusted BN layers into the weights before calculating quantization parameters.
This process effectively "re-centers" the model's activations for the new calibration data without altering the learned weights.
3. Domain Selection via Gram Matrices
To optimize performance, one must select the "best" available out-of-domain dataset. The authors propose using the similarity of Gram matrices (a technique borrowed from style transfer) as a proxy for domain discrepancy. By comparing the "fingerprint" (Gram matrix) of the original data with public datasets, users can identify the most suitable calibration set without ever accessing the sensitive raw images.
Key Results and Insights
The study evaluates 10 tasks across diverse domains, including natural images, medical imaging (X-ray, Ultrasound), and autonomous driving.
- Performance Parity: On ResNet-50 trained on ImageNet, out-of-domain calibration with BN adjustment achieved 75.70% accuracy, nearly matching the in-domain baseline of 75.73%.
- Superiority over Synthetic Data: In specialized domains like medical imaging, real out-of-domain data often provides a more robust signal than synthetic data. For the NIH Chest X-ray task, the proposed method achieved an AUC of 0.761, significantly outperforming the synthetic-data-based ZeroQ (0.718 AUC).
- Efficiency: Unlike synthesis-based methods that require hundreds of backpropagation iterations, BN adjustment only requires a few forward passes, making it much faster and easier to implement.
Conclusion
This pilot study challenges the assumption that in-domain data is necessary for post-training quantization. By leveraging real-world out-of-domain images and simple BatchNorm adjustments, practitioners can achieve high-performance quantization even when the original training data is strictly protected. The correlation between Gram matrix similarity and quantization performance further provides a mathematically grounded way to select optimal calibration datasets.
Summary updated