Data Generation for Hardware-Friendly Post-Training Quantization

Lior Dikstein, Ariel Lapid, Arnon Netzer, Hai Victor Habi

Introduction

Quantization is a critical technique for deploying deep learning models on resource-constrained edge devices. By converting 32-bit floating-point weights and activations into low-precision integers (e.g., 4-bit or 8-bit), models can achieve significant speedups and reduced memory footprints. Post-Training Quantization (PTQ) is particularly attractive as it avoids the high cost of retraining. However, PTQ typically requires a small calibration dataset. In privacy-sensitive domains, access to real training data is often restricted, leading to the development of Zero-Shot Quantization (ZSQ). ZSQ methods generate synthetic data by matching statistics stored in the model's Batch Normalization (BN) layers.

Despite their promise, existing ZSQ methods often fail in "hardware-friendly" scenarios where the entire network, including the final output layer, is quantized. This paper introduces Data Generation for Hardware-Friendly Quantization (DGH), a novel framework that addresses three fundamental gaps in synthetic data generation: the mismatch between local and global statistics, the absence of data augmentation mimicry, and the "last mile" distribution shift at the output layer.

Methodology

The DGH framework incorporates three key innovations to produce high-fidelity synthetic calibration data:

  • Global Statistics Aggregation: Traditional ZSQ methods optimize synthetic images batch-by-batch to match BN statistics. This often leads to "homogenized" data where images lack diversity. DGH employs a memory-efficient algorithm to ensure the entire synthetic dataset matches the model's BN statistics collectively. This relaxation allows individual images to exhibit the variance seen in real datasets.
  • Augmentation-Aware Preprocessing: Real-world models are trained using various data augmentations. DGH integrates a preprocessing pipeline that applies smoothing filters and random augmentations during the generation process. This ensures the synthetic data distribution aligns with the augmented distribution the model encountered during training.
  • Output Distribution Stretching Loss (ODSL): A significant challenge in ZSQ is the lack of BN layers at the network's output. Without guidance, synthetic data often fails to utilize the full dynamic range of the output layer, leading to poor quantization performance. DGH introduces ODSL to "stretch" synthetic activations at the output while maintaining a "natural" range constrained by the preceding BN layer.
t-SNE visualization comparing the diversity of real data, ZeroQ, and the proposed global optimization approach.

Experimental Results

The researchers evaluated DGH on standard benchmarks, including ImageNet-1k for classification and COCO for object detection. In classification tasks, DGH demonstrated superior performance, particularly at low bit-widths. For a ResNet18 model quantized to 4-bit weights and activations (W4A4), DGH achieved 65.64% accuracy. This significantly outperforms the previous state-of-the-art method, IntraQ (47.44%), and even exceeds the accuracy achieved using real calibration data (62.5%) in the same hardware-friendly setup. At extreme 2-bit weight quantization (W2A4), DGH maintained a robust 56.55% accuracy, whereas baseline methods like ZeroQ and IntraQ suffered catastrophic accuracy drops to below 10%.

Output histogram of ResNet-18 showing how DGH aligns synthetic data with the real data distribution compared to standard BN optimization.

For object detection on the COCO dataset, DGH showed massive improvements. On a RetinaNet model, DGH reached a mean Average Precision (mAP) of 32.28, a stark contrast to the 2.60 mAP achieved by ZeroQ.

Discussion and Conclusion

The success of DGH highlights the importance of global statistical matching and addressing the output layer's distribution. By relaxing per-batch constraints, DGH prevents the homogenization of synthetic data, as visualized in t-SNE embeddings where DGH-generated data covers a manifold similar to real data. Furthermore, the ODSL effectively bridges the "last mile" gap, ensuring the final quantization step is calibrated on representative data.

While DGH sets a new benchmark for ZSQ in CNNs, its reliance on Batch Normalization layers presents a limitation for architectures like Vision Transformers (ViTs) that utilize Layer Normalization. Future work will focus on adapting these principles to a broader range of normalization techniques and further reducing the computational overhead of the iterative generation process.

Summary updated

Abstract

Zero-shot quantization (ZSQ) using synthetic data is a key approach for post-training quantization (PTQ) under privacy and security constraints. However, existing data generation methods often struggle to effectively generate data suitable for hardware-friendly quantization, where all model layers are quantized. We analyze existing data generation methods based on batch normalization (BN) matching and identify several gaps between synthetic and real data: 1) Current generation algorithms do not optimize the entire synthetic dataset simultaneously; 2) Data augmentations applied during training are often overlooked; and 3) A distribution shift occurs in the final model layers due to the absence of BN in those layers. These gaps negatively impact ZSQ performance, particularly in hardware-friendly quantization scenarios. In this work, we propose Data Generation for Hardware-friendly quantization (DGH), a novel method that addresses these gaps. DGH jointly optimizes all generated images, regardless of the image set size or GPU memory constraints. To address data augmentation mismatches, DGH includes a preprocessing stage that mimics the augmentation process and enhances image quality by incorporating natural image priors. Finally, we propose a new distribution-stretching loss that aligns the support of the feature map distribution between real and synthetic data. This loss is applied to the model's output and can be adapted to various tasks. DGH demonstrates significant improvements in quantization performance across multiple tasks, achieving up to a 30% increase in accuracy for hardware-friendly ZSQ in both classification and object detection, often performing on par with real data.
BibTeX
@inproceedings{dikstein2025data,
  author = {Lior Dikstein and Ariel Lapid and Arnon Netzer and Hai Victor Habi},
  title = {{Data Generation for Hardware-Friendly Post-Training Quantization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00499},
}
Figure