Abstract
Data-free quantization (DFQ) is essential for deploying deep neural networks on edge devices when original training data is unavailable. However, existing DFQ methods often suffer from performance degradation due to their reliance on statistical correlations in synthetic data, which may include non-causal noise. This paper introduces Causal-DFQ, a framework that leverages causal reasoning to decouple content from style in synthetic data generation. By performing style interventions and aligning model distributions via do-calculus, Causal-DFQ achieves state-of-the-art accuracy on ImageNet benchmarks, sometimes even surpassing models fine-tuned on real-world data.
Introduction
Deep learning models are increasingly deployed on mobile and edge devices, necessitating model compression techniques like quantization. Quantization reduces the precision of weights (e.g., from 32-bit to 4-bit or 6-bit), significantly lowering memory footprint and latency. However, this process typically incurs accuracy loss. While fine-tuning with original training data can recover performance, such data is often inaccessible in privacy-sensitive domains like healthcare or security. Data-Free Quantization (DFQ) addresses this by using synthetic data generated from a pre-trained model's internal statistics.
The authors identify a critical limitation in current DFQ approaches: they focus on statistical correlations rather than causal relationships. Synthetic data often contains "irrelevant factors" (style) such as background or lighting that do not define the object's identity (content). Overfitting to these factors leads to poor generalization in the quantized model. Causal-DFQ aims to solve this by forcing the model to ignore non-causal noise.
Methodology
Causal-DFQ bridges causal inference and model compression through three core innovations:
- Causal Graph Formalization: The data generation process is modeled using a causal graph that distinguishes between Content ($C$) and Style ($S$). Content represents invariant features (e.g., the shape of a car), while Style represents variant environmental factors (e.g., the road background).
- Content-Style-Decoupled Generator: Unlike standard generators that map a single noise vector to an image, this generator accepts separate inputs for content and style. This architecture enables "interventions" on the style component while keeping the content fixed, allowing the system to simulate the same object in diverse environments.
- Style-Intervened Discrepancy Reduction: Using do-calculus, the framework aligns the output distributions of the original floating-point (FP32) model and the quantized model. By ensuring both models respond identically to style interventions, the quantized model is forced to focus on invariant content features. Noise-Contrastive Estimation (NCE) is utilized to measure the similarity between these intervened distributions.
Experiments and Results
The performance of Causal-DFQ was evaluated on standard computer vision benchmarks, including CIFAR-10/100 and ImageNet, as well as object detection tasks on PASCAL VOC 2012.
Causal-DFQ consistently outperformed state-of-the-art (SoTA) methods such as GDFQ, SQuant, and IntraQ. On ImageNet with 6-bit quantization, Causal-DFQ achieved 71.01% accuracy on ResNet-18, approaching the full-precision baseline of 71.47% and notably exceeding the 70.76% achieved by fine-tuning on real data. In 4-bit scenarios, it maintained a significant lead over competitors, demonstrating robustness at low bit-widths (e.g., 72.49% on ResNet-50).
Beyond accuracy, the method is highly efficient. Fine-tuning a 4-bit ResNet via Causal-DFQ required only 8.4 GPU hours, a fraction of the ~30 hours needed for traditional retraining on the full ImageNet dataset.
Conclusion
Causal-DFQ demonstrates that causal reasoning can significantly enhance model compression. By treating the generator as a laboratory for causal interventions, the framework effectively "cleans" synthetic data of non-causal noise. While the current model uses a simplified content-style split, future work could explore more complex causal graphs and extend the methodology to domains like Natural Language Processing (NLP). The success of Causal-DFQ suggests that causality-guided synthetic data can, in some cases, be more effective for model distillation than original real-world datasets.
Summary updated