Dataset condensation compresses a large training set into a small number of synthetic images, for example ten per class, such that a network trained only on them performs as well as possible. The cheapest family of methods, distribution matching, embeds real and synthetic images through randomly initialized networks and pushes the average synthetic feature toward the average real feature. This is fast, but it hits a ceiling. A benchmark study found that once the budget reaches a few hundred images per class, these methods do no better than picking real images at random.
We propose a simple reason for the ceiling and a cheap fix. Matching the class mean uses the same amount of information whether a class gets 10 synthetic images or 500, so the extra images have nothing new to fit. What a larger synthetic set can represent is the shape of each class: its main directions of spread. A set of $n$ points has a covariance of rank at most $n-1$. So the method should match the top $n-1$ directions of the real class covariance, letting the matched rank grow with the budget. Matching only a diagonal (per-coordinate variances) or a fixed Gram matrix (channel-by-channel feature correlations) does not grow in this way.
Core Claim
Our claim is that the high-budget failure of single-level distribution matching is mainly a second-moment failure along off-axis directions. The proposed fix keeps the speed of single-level matching. It adds a low-rank covariance term whose rank is tied to the number of synthetic images per class (IPC).
For one class and one randomly drawn embedding network, let $\mu_r,\mu_s$ be the real and synthetic mean features. Let $\Sigma_r,\Sigma_s$ be their covariances. Let $P_k\in\mathbb{R}^{d\times k}$ hold the top-$k$ eigenvectors of $\Sigma_r$, estimated by a randomized sketch of the current real batch. We minimize
$$ \mathcal{L}=\|\mu_r-\mu_s\|_2^2+\lambda\,\big\|P_k^\top(\Sigma_r-\Sigma_s)P_k\big\|_F^2+\gamma\,\big(\operatorname{tr}[(I-P_kP_k^\top)(\Sigma_r-\Sigma_s)]\big)^2, $$
with $k=\min(\text{IPC}-1,k_{\max})$. The second term matches spread and orientation inside the subspace that the synthetic set is able to span. The third term matches only the total leftover variance outside it, so the synthetic points do not over-spread there. The cost per class is $O(dkb)$ for a batch of $b$ real features. This is linear in the feature width $d$, unlike the $O(C^2)$ Gram term in Decomposed Distribution Matching.
The novelty is not that second moments are matched. Others already match them. What is new is the rule that the matched subspace grows with the budget, and the falsifiable prediction that follows. The advantage over mean-only and diagonal matching should be small at IPC 1 to 10 and should widen at IPC 50 to 500, which is exactly where DC-BENCH finds condensation collapsing to the random-selection baseline.
Relation to Prior Work
Several papers motivate this proposal. Dataset Condensation with Distribution Matching replaced costly two-level gradient unrolling with mean matching and reported a 45x speedup on CIFAR-10. It matches no second moments, and we keep its single-level structure. DC-BENCH documents both the high-budget ceiling and a gap of more than 10% caused by evaluation-time augmentation alone. That second finding constrains our evaluation: every method must use one fixed augmentation protocol.
Other papers already match second moments, and our method must beat them. Improved Distribution Matching and DDM add per-channel variances, and DDM also adds Gram correlations with hand-tuned weights. Neither ties the matched structure to the budget. CovMatch matches the vision–language cross-covariance with pretrained encoders and leaves within-modality covariance unconstrained. We work within one modality, from scratch.
GeoDM attacks the same "flat matching" weakness by moving to curved product manifolds, at the cost of an optimal-transport loss. Our method stays Euclidean and asks how much of GeoDM's gain a rank-aware second moment already recovers.
The diagonal-covariance literature supplies the general lesson and a warning about its limits. Optimal Diagonal Covariance Matching shows that moving from a single scalar variance to a learned diagonal halves diffusion sampling steps (21 down to 10 at comparable quality on ImageNet 256). Its authors point to low-rank structure as the next step. FIVE argues that axis-aligned Gaussians cannot represent curved structure once dimension reaches 3 or more. LCMat and LP-DiF show diagonal approximations working well in condensation and in feature replay. That is a reminder that diagonal may be good enough, and our ablations must allow for this possibility.
Research Program
We will build the objective on top of the public distribution-matching setup used by DM, DDM and the NCFM codebase (gszfwsb/NCFM). Each iteration redraws the random networks and recomputes $P_k$ from a fresh real batch, so the subspace is never fit to a single embedding.
The work has three strands. The first is a diagnostic. For each budget, we measure how much of the real class covariance, in random-network feature space, lies in the top $\text{IPC}-1$ directions. We also measure how well DM's synthetic sets reproduce that subspace, scored by principal angles. This tests whether the ceiling coincides with a growing subspace mismatch.
The second strand is the method itself, together with a rank schedule. We compare a fixed $k$, $k=\text{IPC}-1$, and $k$ set by an energy threshold.
The third strand is an overhead accounting. We report synthesis wall-clock and peak memory next to accuracy, because the point of this regime is to stay near DM's cost of under 20 minutes for CIFAR-10. That cost profile is what separates it from trajectory methods, for which CovMatch cites more than 120 GB of trajectory storage, and from diffusion generation pipelines.
Baselines
The naive reference points are random real-image selection and K-Center selection, evaluated under the DC-BENCH protocol. DC-BENCH reports random selection at 25.67% on CIFAR-10 at IPC 10, against 63.66% for trajectory matching. We found no reported result on Lacuna for random selection at IPC of 200 or more. We will therefore run it ourselves with identical architecture, augmentation and training epochs. At those budgets random selection is the real opponent, and beating it by a margin larger than the seed-to-seed spread counts as breaking the ceiling.
Mean-only DM isolates what the covariance term adds. It reports 43.6% on CIFAR-100 at IPC 50. We rerun it with the same random-network distribution, iteration count and image initialization as our method.
Diagonal-variance matching (the IDM/DDM style) is the key ablation. Improved DM reports 45.1% on CIFAR-100 at IPC 10, and DDM reports gains of up to 4.2% over DM there. We implement a diagonal version inside our own code and give it the same tuning budget for $\lambda$. This removes implementation differences and leaves only the covariance structure as the variable.
A fixed-rank variant uses the same $k$ at every budget. It isolates the rank-tied-to-budget rule from low-rank matching as such.
The strongest existing methods are GeoDM, which reports 74.4% on CIFAR-10 at IPC 10 and 38.0% on CIFAR-100 at IPC 1, and NCFM, which in principle matches all moments through characteristic functions. We run both from released code where available, under our augmentation protocol, with equal GPU-hours. Matching GeoDM within one point at low budgets while exceeding it at IPC 100 or more, at lower synthesis cost, counts as a win. A loss at all budgets would falsify the claim that second moments are the bottleneck.
Why not just…?
Why not just use random or K-Center selection at large budgets? This is the most serious alternative, because DC-BENCH says it already ties condensation at IPC 200 or more. If our method cannot beat random selection at IPC 200 and 500 under matched evaluation, the honest conclusion is that synthesis is not worth its overhead at those budgets. The budget sweep is designed to settle exactly this.
Why not just match diagonal variances, which is cheaper still? Diagonal matching may capture most of the gain. LCMat and LP-DiF both succeed with diagonal statistics. The rank analysis suggests diagonal matching should stop helping as IPC grows, because it does not use the extra directions the larger set can span. The diagonal-versus-rank-matched ablation across IPC tests this directly. If the two curves coincide, we drop the low-rank term.
Why not just use characteristic-function or manifold matching, which captures everything? NCFM and GeoDM are more expressive, and either might dominate. Our bet is narrower: at large budgets, the leading off-axis second moment carries most of the missing signal, at a fraction of the optimal-transport or minimax cost. The equal-compute comparison shows whether that bet holds.
Why not just sample many images from a fast diffusion model? OCM-style samplers reduce step counts. Even so, diffusion pipelines carry large generation costs, such as the 62 hours on 8 GPUs reported for GC-DDPM, and they require a pretrained generator per domain. We include a small diffusion-sampled set at matched IPC on CIFAR-10 as a reference, and we report accuracy per GPU-hour, so this comparison is visible rather than argued away.
Proposed Experiments
We sweep CIFAR-10, CIFAR-100 and TinyImageNet at IPC 1, 10, 50, 100, 200 and 500, plus ImageNet-1K at 64×64 for scale. DM reports 11.4% there at 3.8% compression. Training uses the standard ConvNet, and transfer is checked on ResNet-18 and VGG-16 to probe the transfer failures DC-BENCH describes. Metrics are test accuracy over five seeds, synthesis wall-clock, peak memory, and the principal-angle diagnostic.
The decisive check is the shape of the accuracy gap as IPC grows. We expect our method minus DM, and our method minus diagonal matching, to increase from IPC 10 to IPC 500. We also expect our method to stay above random selection at IPC 200 or more. A flat or shrinking gap would refute the second-moment explanation of the ceiling.
A second check correlates per-class gains with per-class subspace mismatch under DM. Classes whose spread is most off-axis should gain most. Finally, a continual-learning run in the DDM setting, where DDM reports 39.9% against DM's 34.4%, tests whether better-shaped synthetic classes reduce interference between classes.
References
- Decomposed Distribution Matching in Dataset Condensation
- DC-BENCH: Dataset Condensation Benchmark
- Dataset Condensation with Distribution Matching
- Improved Distribution Matching for Dataset Condensation
- CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder
- GeoDM: Geometry-aware Distribution Matching for Dataset Distillation
- Improving Probabilistic Diffusion Models With Optimal Diagonal Covariance Matching
- Beyond Diagonal Covariance: Flexible Posterior VAEs via Free-Form Injective Flows
- Loss-Curvature Matching for Dataset Selection and Condensation
- Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learning
- Zero-shot Generation of Training Data with Denoising Diffusion Probabilistic Model for Handwritten Chinese Character Recognition