Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling

Xiao Cui, Yulei Qin, Xinyue Li, Wengang Zhou, Hongsheng Li, Houqiang Li

Abstract

Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance. Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10.

Introduction

Dataset Distillation (DD) aims to condense massive datasets into tiny synthetic sets—often as small as 10 images per class—while maintaining high model performance. However, real-world data is rarely balanced; it typically follows a "long-tailed" distribution where a few "head" classes dominate and many "tail" classes are underrepresented. Standard DD methods fail in these scenarios because the expert models used to guide distillation become biased toward head classes, and statistical summaries like Batch Normalization (BN) are overwhelmed by frequent samples, leading to poor representation of rare categories.

The Uni-Level Framework

This research introduces a novel Uni-Level Framework that shifts away from the computationally expensive and bias-prone "trajectory matching" approach used in previous state-of-the-art methods like DAMED. Instead, the framework focuses on statistical alignment and fair supervision through three core pillars:

Comparison between the previous DAMED approach and the proposed Uni-Level Framework, highlighting improvements in tail information retention and efficiency.

1. Debiased Expert Models

The framework utilizes two specialized models: an Observer for extracting data statistics and a Teacher for providing soft labels. To prevent these models from ignoring tail classes, the authors implement a "mixture consistency loss" for robustness and a "class-wise debias loss" that dynamically shifts the model's focus toward minority classes during training.

2. Fair BN Recalibration

Standard BN statistics are often corrupted by the high frequency of head-class samples. The proposed method performs a post-hoc forward pass with dynamically adjusted momentum. This ensures that every sample, regardless of class frequency, contributes equally to the statistical anchors used for distillation, preventing representation skew.

3. Confidence-Guided Initialization

Rather than starting with random noise, the synthetic images are initialized through a multi-round selection process. By generating multiple augmentations of real images and using the debiased Teacher model to select the most confident and diverse candidates, the synthetic set starts with a much stronger foundation for capturing essential class features.

Experimental Results

The framework was evaluated on CIFAR-10-LT, CIFAR-100-LT, Tiny-ImageNet-LT, and ImageNet-LT across various Imbalance Factors (IF) and Images Per Class (IPC) budgets.

  • Accuracy Gains: The method achieved a massive performance jump. On CIFAR-100-LT (IPC=10, IF=10), top-1 accuracy improved from 31.5% to 47.1% (+15.6%). Similarly, Tiny-ImageNet-LT saw an 11.8% improvement.
  • Efficiency and Scalability: By avoiding bi-level optimization loops, the method is significantly faster. Distillation time for CIFAR-10-LT dropped from 17 hours to under 45 minutes (a 20x speedup).
  • Memory Footprint: Unlike trajectory-based methods that see memory usage balloon with dataset size, this framework maintains a constant, low memory footprint (e.g., 3.1 GB vs. 15.8 GB for DAMED on CIFAR-100).

Conclusion

By rethinking long-tailed dataset distillation through the lens of statistical alignment rather than trajectory matching, this framework provides a more stable, fair, and efficient path for data condensation. The results demonstrate that debiasing expert models and recalibrating BN statistics are critical for handling real-world data imbalances. Future work may extend this uni-level approach to other challenging domains like Federated Learning, where data imbalance and resource constraints are prevalent.

Summary updated

Abstract

Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance.Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10.
BibTeX
@inproceedings{cui2026rethinking,
  author = {Xiao Cui and Yulei Qin and Xinyue Li and Wengang Zhou and Hongsheng Li and Houqiang Li},
  title = {{Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i5.37341},
}
Figure