TL;DR
DR-DPO reimagines dataset condensation as a preference optimization problem, using a dual-regularized framework to simultaneously maximize class separability and preserve intra-class diversity. By replacing expensive bilevel optimization with a clean, single-level $O(B)$ objective, it beats state-of-the-art methods by up to 5% in accuracy while slashing peak memory usage by up to 10.2× and training time by up to 7.1×.
What Problem Does This Paper Address?
Deep learning relies on massive datasets, making model training computationally expensive and resource-heavy. Dataset condensation aims to solve this by synthesizing a tiny, highly informative set of synthetic images that can train a model to nearly the same accuracy as the full dataset.
However, existing approaches suffer from major bottlenecks. Traditional gradient- and distribution-matching methods ignore explicit class relationships, leading to weak decision boundaries and poor discriminative power. On the other hand, advanced trajectory-matching methods require expensive bilevel optimization (unrolling training loops) that quickly exhausts GPU memory. The authors tackle this dual challenge: how to explicitly optimize both inter-class distinctiveness and intra-class diversity without incurring massive computational overhead.
Methods
The authors introduce DR-DPO (Dual-Regularized Direct Preference Optimization), which frames dataset condensation through the lens of preference learning (popularized by LLM alignment methods like DPO).
The framework consists of two core components optimized jointly in a single forward pass:
- Inter-Class Preference Loss via DPO: Instead of using contrastive losses that require expensive $O(B^2)$ pair mining, DR-DPO uses a log-ratio preference objective. It treats the correct class as the "preferred" option and a randomly sampled incorrect class as the "rejected" option, maximizing the relative margin between synthetic and real data with an efficient $O(B)$ complexity.
- Intra-Class Diversity Loss: To prevent "prototype collapse" (where all synthetic images in a class look identical), the authors introduce a regularizer based on Jensen-Shannon (JS) divergence. It ensures that the diversity between pairs of synthetic samples within the same class meets or exceeds their initial state.
Datasets
The method was evaluated on standard computer vision benchmarks:
- CIFAR-10 and CIFAR-100
- Tiny-ImageNet
- Six ImageNet-1K subsets (ImageNette, ImageWoof, ImageFruit, ImageMeow, ImageSquawk, ImageYellow) scaled to 128×128 resolution to evaluate scalability and performance under class imbalance.
No new datasets were introduced in this paper.
Key Insights and Ideas
- DPO Translates Beautifully to Vision: Direct Preference Optimization isn't just for language models. By framing a synthetic sample's class assignment as a preference over real data margins, the mathematical properties of DPO (like bounded gradients and shift-invariance) provide highly stable optimization for image synthesis.
- Bilevel Optimization is Avoidable: You don't need complex min-max games or unrolled training trajectories to condense data effectively. A well-designed single-level objective can achieve superior accuracy with a fraction of the compute.
- Diversity Needs an Anchor: Simply maximizing intra-class variance can lead to over-dispersion or chaotic images. Calibrating the JS divergence against the initialization state acts as a perfect baseline to maintain natural intra-class variability without excessive spread.
Key Results
- State-of-the-Art Accuracy: DR-DPO consistently outperforms strong baselines like NCFM across all configurations. It achieves top-1 accuracy gains of up to 1.8% on CIFAR-100 and up to 5.0% relative improvement on Tiny-ImageNet.
- Massive Efficiency Gains: On a single NVIDIA A100 GPU, DR-DPO cuts training time per iteration by 1.3× to 7.2× and slashes peak memory by 4.3× to 10.2× compared to trajectory-matching and min-max baselines.
- Cross-Architecture Robustness: Synthetic datasets condensed on a standard ConvNet successfully transfer to AlexNet, VGG-11, and ResNet-18, narrowing the generalization gap typically seen in prior dataset distillation work.
Limitations and Future Work
- Hyperparameter Sensitivity: While robust within a reasonable band, DR-DPO still exhibits sensitivity to its temperature ($\beta$) and regularization weight ($\lambda$) parameters under extreme data scarcity or severe class imbalance.
- Initialization Overhead: The multi-sample composition initialization strategy used to kickstart the process adds an upfront computational cost.
- Future Directions: The authors suggest exploring lighter initialization methods, adaptive hyperparameter schedules, and extending the framework to non-vision modalities (like text or audio) and richer preference modeling structures.
Summary updated