A diffusion model generates images by starting from random noise and denoising it step by step. When we want a model trained on one domain, say photographs of faces, to produce samples from another, say a particular painter's style, from only a handful of example images, the cheapest option is to leave the model's weights alone and nudge each denoising step toward the examples. Recent work does this with Maximum Mean Discrepancy (MMD), a kernel-based distance between two sets of samples. The method computes the distance between the batch being generated and the reference images inside the model's compressed latent space, then follows its gradient. The weak point is the kernel. A fixed Gaussian kernel on raw latents treats every direction equally. In practice the largest distances come from the directions where the pretrained model already varies most, such as pose, background, and lighting. The guidance therefore spends its effort reshuffling nuisance variation instead of changing what actually differs between the domains.
We propose to replace the fixed kernel with the best possible discriminator: a function that separates the reference set from the current batch as sharply as possible, measured relative to how much the pretrained model naturally varies along each feature direction. If this discriminator is linear over an explicit, scalable random-feature map, it has a closed form. It requires one regularized linear solve against a covariance matrix that is computed once, offline. The result keeps the adversarial idea from domain adaptation, a discriminator that seeks the most telling difference, while removing the unstable minimax training. It suppresses redundant high-variance feature directions automatically. Because it works on compact latent tokens with explicit features, its cost grows linearly in the number of reference and generated samples.
Core Claim
Let $\phi(z)\in\mathbb{R}^m$ be an explicit feature map of a latent $z$, built from random or landmark features. Let $\mu_R$ be the mean feature of the reference (target) images, $\mu_G$ the mean feature of the current generated batch, and $\Sigma_S$ the feature covariance of a large pool of unguided samples from the pretrained model. We guide sampling down the gradient of
$$D_\lambda = (\mu_R-\mu_G)^\top(\Sigma_S+\lambda I)^{-1}(\mu_R-\mu_G),$$
where $\lambda>0$ is a ridge parameter. $D_\lambda$ is the squared separation achieved by the optimal linear-in-features discriminator $f(z)=w^\top\phi(z)$, with $w=(\Sigma_S+\lambda I)^{-1}(\mu_R-\mu_G)$. This $w$ maximizes the gap between the reference mean and the batch mean of $f$, divided by the source variance of $f$ plus a norm penalty. As $\lambda\to\infty$, $D_\lambda$ reduces (up to scale) to ordinary MMD with the kernel $\langle\phi(z),\phi(z')\rangle$. The proposal therefore strictly contains fixed-kernel MMD guidance, and $\lambda$ is the single knob that controls how much redundancy is discounted.
We make three claims. First, at finite $\lambda$ the guidance reaches the target distribution faster and with less loss of the source model's diversity than fixed-kernel MMD, because it stops pushing along directions in which the source already varies. Second, the gain comes from whitening in the kernel feature space, not from linear whitening of the latent input. Third, with features computed per latent token and averaged over coarse spatial cells, the method scales to hundreds of references and large batches, where an exact Gram matrix does not.
Where This Sits
This proposal builds on MMD Guidance, which injects gradients of a fixed-kernel MMD between generated latents and a few reference latents into the reverse trajectory of a pretrained latent diffusion model. It reports lower Fréchet Distance (FD), Kernel Distance (KD), and Relative Reproducing Kernel Error (RRKE) than unguided sampling, classifier-free guidance, and classifier guidance. We keep its training-free, latent-space setting and change the discrepancy itself, which that paper leaves fixed.
The adversarial ingredient comes from domain adaptation. Classical adversarial adaptation trains a domain discriminator against a feature extractor, and that minimax game is known to be unstable. Adversarial Spectral Kernel Matching learns the MMD kernel itself adversarially by stochastic gradient descent, on time-series data. We keep the idea of maximizing over the discriminator but solve the inner maximization exactly. This matters because, with ten reference images, a discriminator trained by SGD will overfit. The ridge term is the explicit guard against that.
The redundancy motivation comes from three papers. Greedy eigencomponent selection shows that in deep features the top-variance principal directions often encode background or style nuisance rather than the signal of interest. Bi-level redundancy reduction shows that imbalanced singular spectra drive negative transfer and fixes this with ZCA whitening and orthogonality constraints in convolutional classifiers. PS-VAE shows that diffusion in redundant, high-channel latents wastes capacity, and that a 96-channel compressed latent works better. We move the whitening into the kernel feature space of the guidance signal, where neither of the first two papers operates.
Rethink MMD is a constraint rather than a support. It proves that global MMD minimization blurs class structure, which in generation would appear as collapse toward the reference mean. We therefore measure diversity directly rather than assume our objective avoids this.
The scalability tools are reused, not invented. Random Gegenbauer Features approximate dot-product and Gaussian kernels well, but only in low-to-medium input dimension. Applying features per latent token, with 4 to 96 channels, puts us inside that regime. Accelerated RPCholesky selects Nyström landmarks with the accuracy of sequential pivoted Cholesky at 5x to 50x the speed. Matrix coherence explains why uniform landmarks can fail on structured data. UniCon is the precedent that replacing SGD alignment with a closed-form kernel solve can be faster and more accurate (Flickr30k Recall@1 0.353 vs 0.236 for SGD-CLIP). Our closed form is a ridge solve rather than its singular value decomposition.
Research Program
Building the feature map. For each latent, we take the model's prediction of the clean latent at the current noise level and apply $\phi_{tok}$ to every spatial token. We then average within a coarse spatial grid, at 1×1 for style and 4×4 when layout should matter, and concatenate the results. This gives a mean-embedding set kernel. We will compare three versions of $\phi_{tok}$ at matched feature count $m$: Random Fourier Features for a Gaussian kernel, Random Gegenbauer Features for a dot-product or Gaussian kernel, and Nyström features with landmarks chosen by Accelerated RPCholesky from the source latent pool.
Precomputing the covariance. Offline, we draw $n\approx 10^5$ unguided samples per noise-level bucket and form $\Sigma_S$, which costs $O(nm^2)$. We factor $\Sigma_S+\lambda I$ once, at $O(m^3)$ for $m$ around 4,096.
Guiding the sampler. At each step we compute $w$ against the current batch, which costs $O(Bm+m^2)$ for batch size $B$. We then backpropagate $D_\lambda$ through the denoiser, injecting the gradient as MMD Guidance does. Recomputing $w$ at every step makes this a closed-form adversary that tracks the moving batch.
Analysis. We will characterize how $\lambda$ trades off speed of domain matching against preservation of source variation. We will also test whether the directions most down-weighted by the covariance factor coincide with the nuisance principal directions identified by eigencomponent analysis.
Baselines
The naive reference point is unguided sampling from the pretrained latent diffusion model. It shows how far the target is from the source and what diversity looks like with no intervention.
The strongest existing training-free method is MMD Guidance, run with its fixed product kernel and exact Gram matrix. It is the most direct competitor because it targets exactly this setting: a pretrained latent diffusion model, a few references, and no weight updates. We will run it with the same model, references, number of sampling steps, and seeds as our method. Its guidance scale and kernel bandwidth will get the same tuning budget as our $\lambda$ and scale, selected on held-out reference images. No reported numerical result on the FFHQ, CelebA-HQ, or painting targets was found on Lacuna for MMD Guidance; the file states only that it lowers FD, KD, and RRKE relative to its baselines. We will therefore re-run it ourselves. Beating it means lower FD and KD to held-out target images at equal or better recall of the source model's modes, with the gap holding across shot counts.
The adversarial competitor is classifier guidance with a domain discriminator trained by SGD to separate references from source samples. MMD Guidance already compared against classifier guidance, and it is the natural non-closed-form version of our idea. It gets the same wall-clock budget for discriminator training as we spend on precomputing $\Sigma_S$.
The training-based reference is low-rank fine-tuning of the denoiser on the references at a matched GPU-hour budget. It marks the ceiling a reviewer would expect with weight updates. It is not a method we expect to beat when references are plentiful.
Two ablations isolate the key ingredient. The first sets $\lambda\to\infty$ while keeping our features, which is MMD with the same explicit kernel. It tests whether gains come from whitening or merely from the feature map. The second replaces $\Sigma_S$ with the covariance of the references alone, which tests whether discounting the source model's own variation is what matters.
Why not just…?
Why not just whiten the latents with PCA or ZCA and keep fixed-kernel MMD guidance? This is the most serious threat, and the proposal is designed to settle it. Linear whitening of the input equalizes variance along latent axes. It cannot discount nuisance variation that is nonlinear in the latent, for instance texture statistics that vary jointly across channels. Greedy eigencomponent selection also warns that variance rank is a poor proxy for relevance. We include input-whitened MMD guidance as a baseline at matched tuning. If it matches our method on FD, KD, and the synthetic nuisance test, we will report that the cheap fix suffices and that the kernel-space covariance is unnecessary.
Why not just fine-tune the diffusion model on the references? With hundreds of references and enough compute, fine-tuning may well win, and we expect it to. The regime we target is five to fifty references with no weight updates. There, fine-tuning tends to memorize, and a closed-form ridge-regularized discriminator has a statistical advantage. The shot-count sweep at matched compute will show where the crossover lies.
Why not just train a domain discriminator and use classifier guidance? That is adversarial adaptation in its usual form. With few references, the discriminator overfits and its gradients on noisy latents are unreliable; it also inherits the minimax tuning problems documented for adversarial adaptation. Our discriminator is the exact optimum of a regularized linear problem, recomputed at every step at negligible cost. A direct comparison at matched budget will show whether exactness and regularization matter.
Why not just use exact MMD with more references? The exact Gram matrix grows quadratically with reference and batch size. Our features grow linearly, so we will plot quality and wall-clock time against reference count up to the thousands. If exact MMD with many references closes the quality gap, our contribution narrows to cost, and we will say so.
Proposed Experiments
Synthetic Gaussian mixture. The first decisive check uses a mixture in which source and target differ only along a low-variance direction, while the source has large variance along an unrelated nuisance direction. This is in the spirit of the synthetic mixtures used by MMD Guidance. Fixed-kernel MMD should spend its gradient on the nuisance axis and our guidance should not. We measure error along each axis separately, and failure here would falsify the core claim.
Image domain transfer. We will use pretrained models on FFHQ and CelebA-HQ, guided toward artistic painting targets (Klimt, Franz Marc), with 5, 10, 50, and 500 references. Metrics are FD, KD, and RRKE to held-out target images, following MMD Guidance. Diversity is measured as recall of the unguided model's modes. Source-content preservation is measured as the representation-space similarity between guided and unguided samples drawn with the same seed. We will report a $\lambda$ sweep showing the full curve from whitened guidance to plain MMD. Each experiment runs on both a standard 4-channel latent and a 96-channel PS-VAE-style latent, to test whether the more compact, semantic latent changes the benefit.
Feature-map comparison. We will compare Random Fourier, Gegenbauer, and RPCholesky-Nyström features on approximation error of the token kernel and on downstream FD at fixed $m$. We expect landmark features to matter most on high-coherence latent pools, as the coherence analysis predicts.
Source-free classification (stretch). We will guide a class-conditional, ImageNet-pretrained latent diffusion model toward unlabeled VisDA-C real target images, using class prompts for the twelve categories. The synthesized labeled images will train a classifier that never sees source data. The bar is set by numbers in the file: DVD reports 88.9% on VisDA-C, against 87.8% for NRC++. CAN reports 87.2% with source data available. Matching DVD would be a strong result; landing well below it would still be informative if whitened guidance clearly beats fixed-kernel MMD guidance in the same pipeline, since that comparison isolates our contribution.
References
- MMD Guidance: Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance
- Adversarial Spectral Kernel Matching for Unsupervised Time Series Domain Adaptation
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- Improving Unsupervised Domain Adaptation by Reducing Bi-level Feature Redundancy
- Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing
- Rethink Maximum Mean Discrepancy for Domain Adaptation
- Random Gegenbauer Features for Scalable Kernel Methods
- Embrace Rejection: Kernel Matrix Approximation by Accelerated Randomly Pivoted Cholesky
- Matrix Coherence and the Nystrom Method.
- UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels
- Vicinity-Guided Discriminative Latent Diffusion for Privacy-Preserving Domain Adaptation
- Contrastive Adaptation Network for Unsupervised Domain Adaptation