Image classifiers and detectors trained on photographs fail on sketches, cartoons and paintings, largely because they rely on surface texture that artwork replaces with brushstrokes, hatching or flat color. The standard fix is to train on photos that have been repainted by a style-transfer network. The model then sees "the same cat in many styles" and is pushed to ignore style. This helps, but it has two known costs. First, the strongest methods need several labeled source domains. Second, stripping texture reliance hurts accuracy on photographs, where texture is genuinely useful. We propose a single-source method, meaning it trains on labeled photos only. A frozen photo-trained teacher network supervises a student on stylized copies of each photo. The student must reproduce the teacher's clean-photo features only in the spatial frequency bands that do not change with style, and it is left free everywhere else.
The idea rests on a simple observation that current methods ignore. Arbitrary style transfer with Adaptive Instance Normalization (AdaIN), which re-scales each feature channel to match the mean and variance of a style image, defines style as per-channel statistics. The channel mean is exactly the zero-frequency coefficient of a feature map's spatial frequency transform. Distilling full feature maps from a clean teacher to a stylized student therefore asks the student to undo the very change the augmentation makes. Distilling nothing wastes the teacher's photo knowledge. The right target lies in between: the bands where teacher features vary across images but not across styles.
Core Claim
Our claim is that restricting feature distillation to a data-selected set of style-invariant frequency bands has three effects. It lets a photo-only model close more of the gap to unseen artistic domains than logit-level consistency, full-feature distillation, or stylized training alone. It also retains clean-photo accuracy, which FACT explicitly reports losing.
We further predict that the selected bands will be mostly mid-frequency. This would support, at the level of network features, what FSDR found for input pixels: in synthetic-to-real driving scenes, both the very low and very high DCT bands carry domain-specific variation, while the middle bands carry shapes and boundaries. It would also contradict the rigid "low frequency is content, high frequency is style" split assumed by Frequency Domain Disentanglement for Arbitrary Neural Style Transfer.
Relation to Prior Work
The closest work is FACT. FACT swaps Fourier amplitude spectra between images from different source domains and enforces agreement between a student and a moving-average teacher. Its limits, relative to our setting, are that it needs multiple source domains, operates only on input pixels, distills only class predictions, and loses accuracy on the photo domain of PACS. Cross-domain Ensemble Distillation builds its teacher by averaging predictions across source domains that share a class label, which again requires several labeled domains.
Our method keeps the teacher-student structure of both papers but changes three things. It uses a frozen teacher trained on photos alone. It generates styles with an arbitrary style-transfer network. It applies the distillation loss inside the feature spectrum rather than on output logits.
Several works supply components we reuse. AdaIN and EFDM supply the zero-shot stylizers; EFDM matches the full sorted distribution of each channel rather than only its mean and variance. FSDR supplies the idea of choosing frequency bands by how much they vary across domains. We carry that selection from pixel histograms into distillation targets.
Other works motivate the problem. Detecting People in Artwork with CNNs showed that only early convolutional layers transfer to artwork and that preserving spatial layout is essential. This is why we distill spatial-frequency structure in mid-level layers rather than globally pooled features.
The distillation literature adds both motivation and constraints. Subliminal Learning as Trait-Direction Drift warns that distillation carries unwanted teacher traits along with useful ones. Here the unwanted trait is photographic texture bias, and band masking is our direct response. Following Trust Regularization for RNN language model distillation, which found that dropout and activation noise degrade distillation, we disable dropout in the student during distillation. Work on distillation acting as a regularizer, such as weak-to-strong generalization theory, gives reason to think a student can exceed the teacher on shifted data. We treat this as a hypothesis, not an assumption.
Research Program
Let $x$ be a labeled photo and $y$ a style exemplar drawn from a training style pool. Let $s(x,y)$ be the AdaIN or EFDM stylization of $x$. Let $F_T^\ell$ and $F_S^\ell$ be the teacher's and student's feature maps at layer $\ell$, and let $D$ be the per-channel two-dimensional discrete cosine transform over spatial positions, split into $B$ bands. The student minimizes
$$ \mathcal{L} = \mathrm{CE}\big(S(x)\big) + \mathrm{CE}\big(S(s(x,y))\big) + \lambda \sum_\ell \sum_{b} m_b^\ell \, \frac{\big\| D_b F_S^\ell(s(x,y)) - D_b F_T^\ell(x) \big\|^2}{\nu_b^\ell}. $$
Here CE is cross-entropy with the true label. The mask $m_b^\ell \in \{0,1\}$ selects band $b$ at layer $\ell$. The term $\nu_b^\ell$ is the teacher's mean energy in that band, and dividing by it keeps low bands from dominating, following the coefficient normalization in wavelet-domain super-resolution.
The mask is computed once, before student training, from the frozen teacher alone. For each band we compute a ratio $r_b$. The numerator is the variance of teacher band coefficients across styles for a fixed image. The denominator is the variance across images for a fixed style. We keep the bands whose $r_b$ is below a threshold chosen on a held-out split of training styles, never on test domains.
This definition has two consequences. It automatically excludes the zero-frequency band, since AdaIN changes it by construction. It also excludes whichever high bands the stylizer rewrites.
We will build this at two scales. The first is PACS-style classification with ResNet-18 and ResNet-50. The second is person detection, where the backbone of a COCO-trained Faster R-CNN is distilled on stylized COCO in the manner of StyleCOCO.
"Zero-shot" is enforced strictly. The training style pool, for example Painter by Numbers images, is filtered to remove near-duplicates of any test image. We also report results with the pool restricted to styles visibly unlike each test domain; for instance, no line drawings are allowed when testing on sketches.
Baselines
The naive reference is empirical risk minimization, meaning ordinary cross-entropy training on photos only. It shows how large the gap is.
The strongest single-source augmentation competitor is stylized training with cross-entropy and no teacher, as in Improving Object Detection in Art Images Using Only Style Transfer. That paper reports People-Art AP50 of 0.68 for a ResNet-152 Faster R-CNN trained on StyleCOCO, against a 0.58 baseline. We reproduce it with the same stylizer, the same style pool and the same schedule as our method.
We also run RandConv, which passes images through random convolution filters to scramble texture. It reports 18.09% top-1 accuracy on ImageNet-Sketch, up from 10.28%. We run EFDMix, which mixes feature distributions within a batch, and the domain-adversarial shape-prior method of Kashyap et al., which reports 11% higher ImageNet-R accuracy than a standard model trained on a 200-class ImageNet subset.
For the teacher-student comparison we run a single-source adaptation of FACT. It keeps FACT's logit-level co-teacher consistency, but the amplitude spectra come from style-pool images instead of other source domains.
For context only, the multi-source numbers are 88.15% average PACS accuracy for FACT with ResNet-50, 86.4% for XDED with UniStyle on ResNet-18, and 83.9% and 86.9% for EFDMix on ResNet-18 and ResNet-50. These use more labeled domains than we allow. For the single-source photo-to-other-domains protocol on PACS, no reported result was found on Lacuna. We therefore run every baseline ourselves.
Two ablations isolate the key ingredient. The first uses the same distillation loss with the mask set to all ones, which is full-spectrum feature distillation. The second uses a mask chosen at random with the same number of bands. Every method gets the same backbone, data, stylizer outputs and number of training steps. Each also gets the same tuning budget of twenty hyperparameter trials, selected on a held-out split of training styles.
Beating a baseline means a higher average unseen-domain accuracy (or AP50) over three seeds with non-overlapping confidence intervals, together with a clean-photo accuracy drop of no more than half a point relative to empirical risk minimization.
Why not just…?
Why not just train on stylized photos with cross-entropy? This is the simplest strong alternative, and StyleCOCO shows it helps. However, it gives the network only the label as a target, so nothing tells it which of its photo features to keep. The texture-degradation trade-off noted in the cross-depiction literature suggests it will lose clean-photo accuracy. Our stylized-training baseline, which uses identical stylized data, measures directly whether the teacher term adds anything on either axis.
Why not just enforce logit-level consistency between clean and stylized views, as FACT does? This is the most serious threat. If agreement on class predictions already forces invariance, band selection is decoration. Logits discard spatial structure, which Westlake et al. showed matters for artwork, and the benefit should be largest for detection, where localization depends on intermediate features. The single-source FACT baseline settles this. If it matches our method on both PACS and People-Art, we will report that the frequency component is unnecessary.
Why not just distill the full feature map? Its zero-frequency coefficient is the channel mean that AdaIN rewrites, so full distillation fights the augmentation. We therefore predict it will fall between empirical risk minimization and our method. The all-ones mask ablation tests this. If it does as well as the selected mask, our central mechanism is wrong.
Why not just use the amplitude-swap or RandConv augmentations? Neither method uses a teacher, so neither has a mechanism for retaining photo knowledge. FACT itself reports a photo-domain drop. Clean-photo accuracy is reported alongside every generalization number so that this trade-off is visible.
Proposed Experiments
The primary classification test trains on the PACS photo domain only and tests on art painting, cartoon and sketch. It also trains on ImageNet and tests on ImageNet-Sketch and ImageNet-R, reporting clean ImageNet accuracy alongside. The detection test trains on COCO people and tests on People-Art AP50. Picasso is included as a qualitative check, since no number is reported in the file.
Three checks are decisive, and each could falsify the proposal. First, the selected mask must beat both the all-ones and the random masks on unseen domains while keeping clean-photo accuracy. Second, the bands selected by $r_b$ should concentrate in the middle of the spectrum at mid-level layers. We will report where they fall even if that contradicts the FSDR-inspired prediction. Third, holding the method fixed and swapping EFDM for AdaIN as the stylizer should change which bands are selected. EFDM changes higher-order channel statistics, so we expect its stylizations to perturb more bands, which would show that the mask tracks what the stylizer actually alters rather than a fixed prior.
Finally, we measure shape bias on shape-texture cue-conflict images. We also check whether the student's texture sensitivity is lower than the teacher's, which tests whether the mask blocks transfer of the teacher's texture bias.
References
- FSDR: Frequency Space Domain Randomization for Domain Generalization
- Frequency Domain Disentanglement for Arbitrary Neural Style Transfer
- A Fourier-Based Framework for Domain Generalization
- Cross-domain Ensemble Distillation for Domain Generalization
- Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
- Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization
- Detecting People in Artwork with CNNs
- Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation
- Knowledge Distillation for Recurrent Neural Network Language Modeling with Trust Regularization
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
- Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution
- Improving Object Detection in Art Images Using Only Style Transfer
- Robust and Generalizable Visual Representation Learning via Random Convolutions
- Towards Domain Adversarial Methods to Mitigate Texture Bias