Do Synthetic Calibration Images Hold Up Under Corruption?

Compressing a trained network for a phone or a sensor usually needs a small "calibration set": a few hundred unlabeled inputs passed through the model to set the numeric ranges of low-precision integers, or to decide which weights to delete. When the original training data cannot be shared, practitioners generate these inputs synthetically from the model itself. The usual method is model inversion, which means optimizing random pixels until the network's internal statistics look the way they did during training. Published results show these synthetic sets nearly match real data on clean test accuracy. The compressed model, though, will run on blurry, noisy, badly lit camera input, and nobody has checked whether models calibrated on synthetic images fail sooner on that input than models calibrated on real images.

This note proposes measuring the value of synthetic calibration data in units of real calibration data. We call this an exchange rate: how many real images one synthetic image is worth. We measure it separately on clean and on corrupted test sets. Our hypothesis is that synthetic calibration looks close to free on clean benchmarks but loses much of its value under corruption. We also expect the loss to sit in one place: the activation ranges that calibration sets. We test this by swapping only those ranges between real-calibrated and synthetic-calibrated models, which shows exactly where the problem is and what fixes it.

Core Claim

Synthetic calibration images can look nearly equivalent to real ones on clean data and still be worth much less on corrupted data. The proposal makes this testable as a ratio of two exchange rates and attributes the gap to one mechanism: activation clipping thresholds and pruning masks fitted to unnaturally clean, high-frequency inverted images. Weight-only quantization should be largely immune.

The exchange rate is defined as follows. Fix a compression recipe, a synthetic source $S$, and a test distribution $D$. Let $a_D(S,n)$ be the accuracy of the compressed model on $D$ when calibrated with $n$ synthetic images. Let $n^*_{\text{real}}(D)$ be the smallest number of real calibration images whose compressed model reaches that same accuracy on $D$, interpolated along a measured real-data curve. Then

$$ r_D(S,n) = \frac{n^*_{\text{real}}(D)}{n}, \qquad \text{gap}(S,n) = \frac{r_{\text{clean}}(S,n)}{r_{\text{corrupt}}(S,n)}. $$

A gap near 1 means clean benchmarks report the synthetic data's value honestly. A gap well above 1 means they overstate it.

This borrows the equivalency-ratio idea from On the Equivalency, Substitutability, and Flexibility of Synthetic Data. That paper measured how many synthetic tracking frames replace one real frame during pre-training, finding roughly 30:1 on MOT17. We apply the idea to post-training calibration instead and condition it on the test distribution. The difference matters: that paper's ratio is a single number, while ours is designed to change under shift.

Why This Gap Is Open

Three lines of work motivate the proposal.

The first is generator-free inversion. FRanDI synthesizes images by matching batch-normalization statistics and then distills the compressed model's intermediate features toward the original. It reports 75.9% for ResNet-18 at 4-bit weights and activations on CIFAR-100, and 72.81% for ResNet-50 at 50% unstructured pruning on ImageNet. Its authors describe the synthetic images as "difficult to perceive."

The second is generator-based synthesis with stability regularization. Robustness-Guided Image Synthesis for Data-Free Quantization (RIS) argues that naive synthetic images sit on sharp, fragile regions of the loss surface. It regularizes the generator for stability under input and weight perturbations, reaching 62.16% top-1 for ResNet-50 at 4-bit weights and activations on ImageNet, 10.04 points above GDFQ. It explicitly names out-of-distribution modeling as future work.

Both papers evaluate only on clean splits. So does the language-model study Self-calibration for Language Model Quantization and Pruning. That study found that 16 self-generated sequences beat 128 real C4 samples, and that pruning is far more sensitive than quantization to the choice of calibration data.

The measurement tool comes from Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. It introduces ImageNet-C (15 corruptions at 5 severities), scored by mean corruption error (mCE, which is error averaged over corruptions and normalized to AlexNet = 100), and reports 76.7 for a standard full-precision ResNet-50.

Two results constrain the design. The first is the pruning-versus-quantization asymmetry from self-calibration. If it carries over to vision, the corruption gap should be large for pruning and for activation quantization but small for weight-only quantization, and we pre-register that prediction. The second is Post training 4-bit quantization of convolutional networks for rapid-deployment. It shows weight bias correction can be done offline but activation correction needs real forward passes, and it supplies the analytical clipping method (ACIQ) we use as a data-free reference.

There is also an apparent contradiction between the two literatures. RIS says low-bit quantization is fragile to synthetic data, while self-calibration says quantization is robust to the choice of calibration data. Splitting weight error from activation error, as described below, is how we resolve it rather than assume either view.

Research Program

The study has three parts.

Exchange-rate curves. For each compression recipe we build real-data curves by calibrating with 8 to 1024 held-out ImageNet training images. We then calibrate with synthetic sets of matched size and read off $r_{\text{clean}}$ and $r_{\text{corrupt}}$. Corrupted accuracy is measured per ImageNet-C category (noise, blur, weather, digital). Stability is measured as prediction flip rate on ImageNet-P, a benchmark of image sequences with gradual perturbations.

The models are ResNet-18, ResNet-50 and MobileNetV2, matching the RIS and FRanDI settings. The compression recipes are:

  • 8-bit weights and activations;
  • 4-bit weights and activations;
  • 4-bit weights with full-precision activations;
  • 50% unstructured pruning;
  • 2:4 semi-structured sparsity.

The synthetic sources span the quality range: random Gaussian inputs, batch-norm inversion in the FRanDI style, a GDFQ-style generator, and RIS. Using the same model weights throughout means only the calibration data changes.

Attribution by component swapping. For a model calibrated on synthetic data, we transplant exactly one component from its real-calibrated counterpart and measure how much of the corruption gap closes. The transplanted component is one of three: the activation clipping thresholds, the recalibrated batch-norm statistics, or the pruning mask. If our mechanism is right, swapping clipping thresholds alone recovers most of the gap for activation quantization, and swapping masks recovers it for pruning.

We also record per-layer activation range statistics on corrupted inputs, to test whether noise and blur push activations outside ranges fitted to inverted images. This is the plain-language version of RIS's claim that synthetic data occupies a different region of the loss surface. The difference is that we measure it on the deployment distribution rather than on the calibration distribution.

Moderator: how flat the trained model's loss surface is. Training Dynamics Impact Post-Training Quantization Robustness shows, for language models, that averaging weights across checkpoints puts the model in a flatter region of the loss surface and makes it much more tolerant of quantization. We repeat the main grid on ResNet-50 checkpoints with and without weight averaging. The question is whether flatter models shrink the corruption gap, which would mean the problem can be fixed on the training side instead of the data side.

Baselines

Naive reference. This is round-to-nearest quantization with min/max ranges taken from random Gaussian inputs, and magnitude pruning without calibration. It sets the floor that every exchange rate is measured against.

Real in-distribution calibration. Held-out ImageNet training images serve as the upper reference and define the denominator of every exchange rate.

Real out-of-domain calibration. We also calibrate with images from PASCAL VOC 2012. This reflects what a practitioner without the training data might actually do: grab some other public images.

Strongest existing synthetic method: RIS. RIS reports 62.16% top-1 for ResNet-50 at 4-bit weights and activations on ImageNet. We found no reported ImageNet-C or ImageNet-P result for RIS or FRanDI on Lacuna, so those numbers are what this study produces. For pruning, the strongest reported synthetic method is FRanDI at 72.81% (ResNet-50, 50% unstructured, ImageNet).

Fairness controls. Every calibration source uses the same quantizer, the same pruning criterion, the same number of samples, the same 500-iteration feature-regression budget that FRanDI reports, and the same hyperparameter search. We reproduce RIS's clean number before trusting our pipeline. A synthetic method "beats" another on our question if its $r_{\text{corrupt}}$ is higher at equal clean accuracy, with confidence intervals from three calibration seeds.

Ablations. The component swaps described above are the main ablation, because they isolate the claimed mechanism. A second ablation adds corruption augmentation to the synthetic calibration images, described under the alternatives below.

Why not just…?

Why not just calibrate on a few public web images? Often this is legally impossible. Medical and proprietary settings are exactly where FRanDI and self-calibration motivate data-free compression. Even when it is allowed, self-calibration found that external data hurts pruning. We include out-of-domain PASCAL VOC calibration as a measured baseline rather than dismissing it. If it matches in-distribution real data under corruption, that finding is itself useful for practitioners.

Why not just add corruptions to the synthetic calibration images? This is the real threat to the proposal. It may close most of the gap, so we test it directly instead of arguing around it. We augment the synthetic sets with noise and blur corruptions only, then evaluate on the held-out weather and digital categories.

  • If augmentation recovers the held-out gap, the conclusion is that a cheap fix suffices. The contribution then becomes the exchange-rate measurement that justifies the fix.
  • If it helps only on the corruptions it saw, the gap concerns how ranges are fitted, not which corruptions were covered. That supports the component-level account.

Why not just use analytical clipping that needs no data? ACIQ derives clipping thresholds in closed form from assumed bell-shaped distributions, and the 4-bit post-training paper notes it cannot correct activation shifts without forward passes. We run ACIQ-only clipping as a baseline. If its corrupted accuracy equals that of synthetic calibration, synthetic data adds nothing under shift, and that would be a sharp negative result.

Why not just use RIS, which already optimizes for stability? RIS optimizes stability under small perturbations around clean synthetic images. That is not the same as the large, structured shifts in ImageNet-C. Whether RIS's 10-point clean gain carries over under corruption is exactly what its gap ratio will show.

Proposed Experiments and Decisive Checks

The primary experiment is the full grid of exchange-rate curves on ImageNet and ImageNet-C, across three architectures, five compression recipes and four synthetic sources, with mCE and ImageNet-P flip rate as secondary metrics.

Three checks decide the outcome:

  • Existence of the gap. If the gap ratio stays near 1 for batch-norm inversion and RIS at 4-bit weights and activations, the core claim is false, and we report that clean benchmarks are trustworthy for calibration.
  • Location of the gap. If swapping clipping thresholds does not recover most of the gap while swapping masks or batch-norm statistics does, the mechanism is wrong and we report the component that actually matters.
  • Pre-registered asymmetry. The weight-only quantization gap should be markedly smaller than the gaps for pruning and for activation quantization. Seeing this would carry the self-calibration asymmetry from language models over to vision and resolve the RIS-versus-self-calibration disagreement.

The weight-averaging moderator and the held-out-corruption augmentation test then determine whether the remedy belongs on the training side, the data side, or both.