Causal-DFQ: Causality Guided Data-free Network Quantization

Yuzhang Shang, Bingxin Xu, Gaowen Liu, Ramana Rao Kompella, Yan Yan

Abstract

Data-free quantization (DFQ) is essential for deploying deep neural networks on edge devices when original training data is unavailable. However, existing DFQ methods often suffer from performance degradation due to their reliance on statistical correlations in synthetic data, which may include non-causal noise. This paper introduces Causal-DFQ, a framework that leverages causal reasoning to decouple content from style in synthetic data generation. By performing style interventions and aligning model distributions via do-calculus, Causal-DFQ achieves state-of-the-art accuracy on ImageNet benchmarks, sometimes even surpassing models fine-tuned on real-world data.

Introduction

Deep learning models are increasingly deployed on mobile and edge devices, necessitating model compression techniques like quantization. Quantization reduces the precision of weights (e.g., from 32-bit to 4-bit or 6-bit), significantly lowering memory footprint and latency. However, this process typically incurs accuracy loss. While fine-tuning with original training data can recover performance, such data is often inaccessible in privacy-sensitive domains like healthcare or security. Data-Free Quantization (DFQ) addresses this by using synthetic data generated from a pre-trained model's internal statistics.

The authors identify a critical limitation in current DFQ approaches: they focus on statistical correlations rather than causal relationships. Synthetic data often contains "irrelevant factors" (style) such as background or lighting that do not define the object's identity (content). Overfitting to these factors leads to poor generalization in the quantized model. Causal-DFQ aims to solve this by forcing the model to ignore non-causal noise.

Methodology

Causal-DFQ bridges causal inference and model compression through three core innovations:

  1. Causal Graph Formalization: The data generation process is modeled using a causal graph that distinguishes between Content ($C$) and Style ($S$). Content represents invariant features (e.g., the shape of a car), while Style represents variant environmental factors (e.g., the road background).
  2. Content-Style-Decoupled Generator: Unlike standard generators that map a single noise vector to an image, this generator accepts separate inputs for content and style. This architecture enables "interventions" on the style component while keeping the content fixed, allowing the system to simulate the same object in diverse environments.
  3. Style-Intervened Discrepancy Reduction: Using do-calculus, the framework aligns the output distributions of the original floating-point (FP32) model and the quantized model. By ensuring both models respond identically to style interventions, the quantized model is forced to focus on invariant content features. Noise-Contrastive Estimation (NCE) is utilized to measure the similarity between these intervened distributions.
The Causal-DFQ framework consists of a Content-Style-Decoupled Generator and a Style-Intervened Alignment module.

Experiments and Results

The performance of Causal-DFQ was evaluated on standard computer vision benchmarks, including CIFAR-10/100 and ImageNet, as well as object detection tasks on PASCAL VOC 2012.

Causal-DFQ consistently outperformed state-of-the-art (SoTA) methods such as GDFQ, SQuant, and IntraQ. On ImageNet with 6-bit quantization, Causal-DFQ achieved 71.01% accuracy on ResNet-18, approaching the full-precision baseline of 71.47% and notably exceeding the 70.76% achieved by fine-tuning on real data. In 4-bit scenarios, it maintained a significant lead over competitors, demonstrating robustness at low bit-widths (e.g., 72.49% on ResNet-50).

Radar charts comparing Causal-DFQ against SQuant and IntraQ across various architectures, showing superior performance in accuracy.

Beyond accuracy, the method is highly efficient. Fine-tuning a 4-bit ResNet via Causal-DFQ required only 8.4 GPU hours, a fraction of the ~30 hours needed for traditional retraining on the full ImageNet dataset.

Conclusion

Causal-DFQ demonstrates that causal reasoning can significantly enhance model compression. By treating the generator as a laboratory for causal interventions, the framework effectively "cleans" synthetic data of non-causal noise. While the current model uses a simplified content-style split, future work could explore more complex causal graphs and extend the methodology to domains like Natural Language Processing (NLP). The success of Causal-DFQ suggests that causality-guided synthetic data can, in some cases, be more effective for model distillation than original real-world datasets.

Summary updated

Abstract

Model quantization, which aims to compress deep neural networks and accelerate inference speed, has greatly facilitated the development of cumbersome models on mobile and edge devices. There is a common assumption in quantization methods from prior works that training data is available. In practice, however, this assumption cannot always be fulfilled due to reasons of privacy and security, rendering these methods inapplicable in real-life situations. Thus, data-free network quantization has recently received significant attention in neural network compression. Causal reasoning provides an intuitive way to model causal relationships to eliminate data-driven correlations, making causality an essential component of analyzing data-free problems. However, causal formulations of data-free quantization are inadequate in the literature. To bridge this gap, we construct a causal graph to model the data generation and discrepancy reduction between the pre-trained and quantized models. Inspired by the causal understanding, we propose the Causality-guided Data-free Network Quantization method, Causal-DFQ, to eliminate the reliance on data via approaching an equilibrium of causality-driven intervened distributions. Specifically, we design a content-style-decoupled generator, synthesizing images conditioned on the relevant and irrelevant factors; then we propose a discrepancy reduction loss to align the intervened distributions of the pre-trained and quantized models. It is worth noting that our work is the first attempt towards introducing causality to data-free quantization problem. Extensive experiments demonstrate the efficacy of Causal-DFQ. The code is available at https://github.com/42Shawn/Causal-DFQ.
BibTeX
@inproceedings{shang2023causal,
  author = {Yuzhang Shang and Bingxin Xu and Gaowen Liu and Ramana Rao Kompella and Yan Yan},
  title = {{Causal-DFQ: Causality Guided Data-free Network Quantization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure