TL;DR
This paper introduces Expert Subspace Projection (ESP), a novel method for dataset condensation that significantly reduces memory and computation costs while preserving long-range training dynamics. ESP achieves this by projecting gradients onto a learned subspace of the original dataset's training trajectory, outperforming previous state-of-the-art methods in synthetic data generation for downstream tasks.
What Problem Does This Paper Address?
The ever-increasing size of datasets in deep learning poses significant challenges for storage, computation, and training time. While dataset condensation aims to create smaller, synthetic datasets that mimic the performance of the original, large datasets, existing methods like trajectory matching, though effective, are computationally expensive due to the need to unroll gradients across many training steps. This unrolling leads to high memory usage, often forcing the use of sub-sampled synthetic data, which can introduce bias and degrade performance. The core problem is to distill the essential knowledge from large datasets into compact synthetic ones efficiently, without sacrificing performance or introducing biases.
Methods
The proposed method, Expert Subspace Projection (ESP), tackles the memory and computation bottleneck of trajectory matching. Instead of directly mimicking the entire training trajectory of a model on the real dataset, ESP leverages the subspace spanned by this trajectory.
Here's a breakdown of the key components:
- Expert Trajectory: The paper first trains a model on the original dataset (Dreal) to generate an "expert trajectory," which is a sequence of weight snapshots. This trajectory captures the long-range dynamics of the training process.
- Expert Subspace: An "expert subspace" is constructed from this expert trajectory. This subspace is a lower-dimensional representation that effectively captures the essential information of the training dynamics.
- Projected Gradient Descent (Inner Optimization): When training with the synthetic dataset (Dsyn), ESP uses projected gradient descent. Instead of updating the model's parameters directly with the computed gradient, the gradient is first projected onto the expert subspace. This ensures that the synthetic model's training stays within the learned subspace, effectively mimicking the long-range dynamics without the need for extensive gradient unrolling.
- Projection Loss (Outer Optimization): A "projection loss" is introduced to penalize any deviation of the gradient from the expert subspace. This loss, combined with a distribution matching loss (to align feature distributions between real and synthetic data), forms the outer optimization objective for updating the synthetic dataset.
This approach significantly reduces memory requirements because the inner optimization only involves a single gradient step, unlike trajectory matching which requires unrolling gradients over multiple steps.
Datasets
The paper evaluates its method on several standard image classification benchmarks:
- CIFAR10/100: Widely used datasets with 32x32 pixel images in 10 and 100 classes, respectively.
- TinyImageNet: A downsized version of ImageNet, often used for benchmarking deep learning models, with 64x64 pixel images across 200 classes.
- SVHN (Street View House Number): A dataset of house numbers from Google Street View, useful for digit recognition tasks.
The authors pre-compute expert trajectories on ConvNet-3 for each dataset to expedite the condensation process.
Key Insights and Ideas
- Subspace is Sufficient: The core insight is that the subspace spanned by the expert trajectory, rather than the entire trajectory itself, is sufficient to capture the crucial long-range training dynamics. This allows for significant computational and memory savings.
- Gradient Projection as a Proxy: Projecting the gradient onto this expert subspace acts as an effective proxy for mimicking the complex, multi-step gradient unrolling required by trajectory matching.
- Memory Efficiency Enables Full Dataset Training: By drastically reducing memory overhead, ESP allows for training on the complete synthetic dataset without resorting to batching or slicing. This avoids potential biases introduced by sub-sampling and leads to better performance.
- Complementarity of Losses: The combination of the projection loss (for long-range dynamics) and distribution matching loss (for static feature alignment) is crucial for achieving state-of-the-art results. The projection loss constrains higher-order information, while distribution matching captures lower-level texture and visual details.
Key Results
- State-of-the-Art Performance: ESP achieves state-of-the-art results on SVHN, CIFAR10, CIFAR100, and TinyImageNet datasets.
- Outperforms Trajectory Matching: On CIFAR10 with 1 Image/Class, ESP outperforms trajectory matching by 16.7%.
- Significant Improvement over SOTA: ESP surpasses the previous state-of-the-art method by 3.2% on CIFAR10.
- Memory Efficiency: ESP demonstrates significantly lower memory consumption compared to trajectory matching, with memory usage remaining largely unaffected by the student trajectory length or the number of images per class.
- Cross-Architecture Generalization: The synthetic datasets generated by ESP show robust performance across different network architectures (ResNet, DenseNet, EfficientNet), indicating good generalization capabilities.
- Batch Size Advantage: Due to its memory efficiency, ESP can train on the full synthetic dataset, leading to better performance than methods that require batching.
Limitations and Future Work
- Bi-level Optimization: ESP still operates within a bi-level optimization framework. The authors acknowledge that extending this to very large datasets with high-resolution images remains a challenge, similar to other bi-level methods.
- Future Directions: Promising avenues for future work include disentangling the outer and inner optimization steps and approximating the inner optimization using a convex proxy model to further reduce computational requirements and memory usage.
Summary updated