Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling

Xiao Cui, Yulei Qin, Xinyue Li, Wengang Zhou, Hongsheng Li, Houqiang Li

Abstract

Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance. Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10.

Introduction

Dataset Distillation (DD) aims to condense massive datasets into a tiny "synthetic" set (e.g., 10 images per class) that retains the performance of the original data. While DD has seen success on balanced datasets, it often fails on long-tailed distributions, where a few "head" classes dominate and many "tail" classes have sparse representation.

Existing methods like DAMED face two primary hurdles in long-tailed scenarios:

  1. Representation Bias: Expert models used to guide distillation are biased toward head classes, resulting in poor synthetic data for the tail.
  2. Computational Overhead: State-of-the-art bi-level trajectory matching is slow and memory-intensive, making it impractical for large-scale datasets.
Comparison of the proposed framework against the DAMED baseline.

Methodology

The proposed Uni-Level Framework shifts away from complex trajectory matching toward statistical alignment. This approach is built on three pillars:

1. Debiased Expert Models

The framework utilizes two models trained on the original data: an observer for feature statistics and a teacher for labeling. To prevent these models from ignoring tail classes, the authors employ a mixture consistency loss and a dynamically-weighted rebalancing strategy.

2. Fair BN Statistics Recalibration

Standard Batch Normalization (BN) statistics are skewed by class frequency. The authors implement a post-hoc recalibration step using a full forward pass with a dynamic momentum coefficient. This ensures that every sample, regardless of class frequency, contributes equally to the global statistics used for distillation.

3. Confidence-Guided Multi-Round Initialization

Instead of random initialization, the teacher model scores multiple augmentations of real images. The framework progressively selects the most "confident" and diverse samples to initialize the synthetic set, ensuring a high-quality starting point even for rare classes.

The detailed architecture of the proposed RLDD framework, showing expert training and synthesis.

Experimental Results

The framework was evaluated on CIFAR-10-LT, CIFAR-100-LT, Tiny-ImageNet-LT, and ImageNet-LT.

  • Performance Gains: On CIFAR-100-LT (IPC=10, IF=10), the method improved top-1 accuracy by 15.6% over the previous SOTA. Tiny-ImageNet-LT saw an 11.8% improvement.
  • Efficiency: The method is approximately 20x faster than DAMED. For CIFAR-10-LT, runtime was reduced from ~17 hours to just ~42 minutes.
  • Scalability: Unlike trajectory-based methods, this approach maintains constant, low GPU memory usage, regardless of the number of images per class (IPC).
  • Generalizability: Synthetic data generated with a ConvNet-3 successfully trained VGG, ResNet, and AlexNet models, demonstrating cross-architecture robustness.

Conclusion

By rethinking long-tailed dataset distillation through the lens of statistical alignment rather than trajectory matching, this work achieves significant improvements in both accuracy and efficiency. The transition to a uni-level framework avoids the biases inherent in previous methods and provides a scalable solution for real-world, imbalanced datasets. Future work may extend this framework to multi-domain or federated learning environments where data imbalance is prevalent.

Summary updated

Abstract

Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance. Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10. Codes are available at https://github.com/2018cx/RLDD.
BibTeX
@inproceedings{cui2026rethinking,
  author = {Xiao Cui and Yulei Qin and Xinyue Li and Wengang Zhou and Hongsheng Li and Houqiang Li},
  title = {{Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i5.37341},
}
Figure