Distilling Toward the Deployment Domain: Data-Free Students Synthesized from Target Statistics

When a large image classifier is shipped to a phone or a car, two things usually go wrong at once. The original training data cannot be shared, so the small "student" model has to be trained by data-free knowledge distillation. In this setting a generator invents images that make the frozen "teacher" respond as it did during training, and the student learns to copy the teacher on those invented images. The deployment camera also sees fog, blur, sensor noise or a different visual style that neither model was trained on. Today these problems are solved separately. Fast data-free distillation methods make synthesis cheap but only ever test the student on clean, in-distribution images. Test-time adaptation methods repair a deployed model under shift but never produce a compressed one. Methods that do both are slow and stack extra networks on top of an already expensive synthesis loop.

This note proposes a single cheap step that joins them. Take a few hundred unlabeled images from the deployment domain and use them only to measure the teacher's internal activation statistics there. Then steer a pre-trained, quickly adaptable generator to invent images whose statistics match those target measurements rather than the source ones. The student is distilled on synthetic data that already "looks like" the deployment domain to the teacher, and it ships as a static compressed model with no online updates. Because the generator is meta-trained once and only fine-tuned for a few steps per new domain, the per-domain synthesis cost should be minutes rather than hours.

Core Claim

Almost every inversion method matches the teacher's stored batch-normalization statistics. These are the per-channel means and variances that the teacher recorded on its source training data. Batch normalization is a layer that rescales each channel using such stored averages. Test-time adaptation work shows that recomputing these averages on a small amount of unlabeled target data recovers much of the accuracy lost under shift. The Norm Must Go On (DUA) updates them on the fly with no backpropagation and cuts CIFAR-10-C severity-5 error from 49.2% to 26.8% within 100–200 samples. Our claim is that the recomputed target statistics can serve as a different inversion target. Swapping them in turns a source-domain synthesizer into a target-domain synthesizer at almost no extra cost. Concretely, for a synthetic batch $x$ with sampled labels $y$, the generator minimizes

$$ \mathcal{L}(x) = \mathrm{CE}\big(T(x), y\big) + \alpha \sum_{l} \Big( \lVert \mu_l(x) - \hat{\mu}_l^{\text{tgt}} \rVert^2 + \lVert \sigma_l^2(x) - \hat{\sigma}_l^{2,\text{tgt}} \rVert^2 \Big), $$

where $T$ is the teacher, $\mu_l(x)$ and $\sigma_l^2(x)$ are the channel means and variances of the batch at normalization layer $l$, $\hat{\mu}_l^{\text{tgt}}, \hat{\sigma}_l^{2,\text{tgt}}$ are the same quantities estimated from the unlabeled target images, and $\alpha$ weights the statistics term.

The teacher used for the cross-entropy and the distillation targets is itself recalibrated to the target statistics. Early-layer statistics encode low-level appearance such as noise energy and blur, and deeper ones encode style. Matching them should therefore push synthetic images toward target-like appearance while the teacher's class semantics keep them labeled. The efficiency half of the claim comes from Up to 100x Faster Data-Free Knowledge Distillation (FastDFKD). That method meta-trains a generator, using a first-order Reptile-style loop, to capture reusable low-level features so that new batches need only 2–10 adaptation steps. It frames those features as shared across synthesis tasks. We treat each deployment domain as a new task for the same meta-generator, which FastDFKD never tested, since it evaluated only clean i.i.d. benchmarks.

Relation to Prior Work

Two lines of work already combine data-free distillation with shift, and both pay heavily for it. AuG-KD trains an inversion generator, then a separate network that maps target images into the teacher's latent space, then a mixup curriculum. It reaches 58.8% on Office-31 (Webcam+DSLR→Amazon) and 21.2% on Office-Home (CPR→A). We keep its setting of a frozen teacher plus unlabeled target images but replace its two extra trained stages with a statistics swap and a few generator steps. Towards Data-Free Domain Generalization (DEKAN) inverts multiple source teachers to build domain-agnostic bridge images, reaching 83.9% on PACS. Its cost grows with the number of teachers, and it deliberately ignores the target domain. We use target information, but only first- and second-order activation statistics, never labels or stored target images in the student pipeline.

A third paper, Momentum Adversarial Distillation, handles a different shift: the drift of the generator's own output during adversarial distillation. It does so by adding an averaged generator copy, at about 40% extra training time. It constrains our design, because adversarial "hard sample" generation is exactly what we avoid. Mosaicking to Distill distills from uncurated proxy images via a four-player game. It shows that real out-of-domain textures help, but it needs an external corpus and is sensitive to patch size.

The efficiency methods supply our tools and also motivate a diagnostic. Small Scale Data-Free Knowledge Distillation (SSD-KD) trains on 10% of the usual synthetic volume by prioritizing samples where teacher and student disagree, finishing CIFAR-10 in about an hour. MUSE synthesizes at reduced resolution and reaches 47.12% on ImageNet (ResNet-50→MobileNetV2, 10% data). Neither reports corruption robustness. Benchmarking Neural Network Robustness to Common Corruptions gives us ImageNet-C and CIFAR-style corruption protocols, and its results on texture-sensitive models make it plausible that boundary-focused, low-volume synthesis yields brittle students.

Finally, Quantifying Cross-Domain Knowledge Distillation in the Presence of Domain Shift proves, for linear models, that under severe shift the student should sometimes weight the teacher negatively. We read this as motivation: an unshifted teacher can be actively harmful. Our response is to move the teacher toward the target before imitating it, rather than to subtract it. If our method fails mainly at the most severe shifts, that result points toward their anti-learning remedy.

Research Program

The work has three parts. The first is a diagnostic study that no paper in the file reports: how students from FastDFKD, SSD-KD and MUSE degrade under corruption compared with students distilled on real source data at matched clean accuracy. This tests whether cheap synthesis buys speed at the cost of robustness.

The second part builds the target-statistics synthesizer. We meta-train a FastDFKD-style generator once on the source statistics. For each target domain we estimate target statistics from $n$ unlabeled images, recalibrate the teacher, and adapt the generator for a fixed small number of steps under the loss above. We then distill a student on the resulting images, optionally with SSD-KD's disagreement-prioritized replay to keep volume small. The student's own normalization statistics are frozen after distillation, so the shipped model works at batch size 1.

The third part analyzes when this works. We measure whether synthetic images actually acquire target-domain low-level properties: the per-layer statistic gap to real target images, and frequency spectra for noise and blur corruptions. We also check whether the gains track how much of the shift lives in normalization statistics. Recalibration methods help less on style shift than on corruption, a distinction visible in the gap between Tent and BN-only adaptation on ImageNet-C. The method also depends on batch normalization, so vision transformers are explicitly out of scope for this proposal.

Baselines

The naive reference is a student distilled data-free from the unmodified source teacher with the same meta-generator and synthesis budget, then deployed as is. It measures how much shift hurts when nothing is done.

The strongest competitor for the static-student question is distill-then-recalibrate. We take that same student and, offline, recompute its batch-normalization statistics on the same $n$ unlabeled target images, as DUA does for a deployed model. It costs essentially nothing and yields an equally static artifact, so our method must beat it by a margin that justifies the extra few minutes of synthesis. On CIFAR-10-C severity 5, DUA reports 26.8% error when applied to the full model. No reported result was found on Lacuna for DUA applied to a data-free distilled student, so we will establish that number ourselves.

The online counterpart is Tent applied to the distilled student during deployment. Tent minimizes prediction entropy on the scale and shift parameters of normalization layers and reports 44.0% ImageNet-C error with a ResNet-50 teacher-scale model, against 49.9% for test-time BN recalibration. We run it with batch sizes of 1, 16 and 64 and report its extra inference compute, since our student needs none. T3A, which re-estimates classifier prototypes from confident test predictions without backpropagation and reports 80.0% on VLCS, serves as a second backpropagation-free online baseline on the Office-style shifts. For out-of-domain distillation we compare against AuG-KD on its own Office-31 and Office-Home splits, where it reports 58.8% and 21.2%.

Two ablations isolate the key ingredient. The first, recalibrated-teacher-only, recalibrates the teacher's statistics for the distillation targets but keeps source statistics in the synthesis loss. It shows whether moving the synthetic images, rather than only the labels, matters. The second runs FastDFKD-style synthesis from scratch on target statistics with no meta-initialization, and shows what amortization buys in per-domain cost.

All methods use the same teacher, student architecture, number of target images, total GPU-hour budget for synthesis plus distillation, and an equal hyperparameter search of twenty configurations tuned on a held-out corruption family. Beating the strongest baseline means at least two points lower error averaged over corruption types at severity 5, with per-domain synthesis cost under one tenth of from-scratch synthesis.

Why not just…?

Why not just distill once and recalibrate the student's normalization statistics on the target images? This is the real threat, and the proposal is built so it can win. It adjusts only the student's per-channel scale and shift. A small student trained on source-like synthetic images may never have learned features that separate classes under heavy noise or blur, and no rescaling creates them. The head-to-head comparison at matched target-image count is our primary experiment. If the gap is under a point across CIFAR-10-C and ImageNet-C, we will conclude that target-aware synthesis is not worth its cost and report that.

Why not just run Tent or DUA on the deployed student? These adapt online, so they need batches. Their known small-batch weakness motivated memory-based fixes like MemBN. Entropy minimization can also lock in confident mistakes. They also add backpropagation or statistic updates to an edge device that may run a compiled, frozen model. We still report them, because if Tent at batch size 64 matches our static student, the advantage reduces to deployment convenience, and we will say so.

Why not just distill on the few hundred target images directly, using teacher labels? This needs no generator at all. With 100–200 images, though, a student sees each class only a handful of times. MosaicKD's results suggest real images help most as texture sources, not as a full training set. We include plain distillation on augmented target images, and a hybrid that mixes them with our synthetic images, at every value of $n$. If direct distillation wins once $n$ reaches about 1,000, that marks the regime where synthesis is unnecessary.

Proposed Experiments

On CIFAR-10 with a ResNet-34 teacher and ResNet-18 student, we evaluate on all 15 CIFAR-10-C corruptions at severity 5, with $n \in \{10, 50, 200, 1000\}$ target images per corruption. On ImageNet with ResNet-50 → MobileNetV2, following MUSE's setup, we report ImageNet-C mean corruption error. For style shift we use Office-31 and Office-Home on AuG-KD's splits. Each run records target error, per-domain synthesis GPU-minutes, and student inference latency.

The decisive checks are four. First, our student must beat distill-then-recalibrate at equal $n$. Second, the meta-initialized generator must match from-scratch target synthesis within about one point at a tenth of the cost. Third, the statistic gap between synthetic and real target images must shrink during adaptation. Fourth, the source-statistics ablation must lose most of the gain, which would show that moving the images, not just relabeling them, is what matters.

A negative outcome is also informative. If gains concentrate on noise and blur and vanish on Office-Home art, then normalization statistics carry corruption shift but not style shift, and the extension needs feature-level targets.