Distribution-aware Dataset Distillation for Efficient Image Restoration

Zhuoran Zheng, Xin Su, Chen Wu, Xiuyi Jia

Abstract

Training state-of-the-art image restoration models requires massive datasets and significant computational resources. We present TripleD (Distribution-aware Dataset Distillation), a framework that synthesizes a representative "mini-dataset" (1–5% of the original size). TripleD enables training high-performance models on consumer-grade hardware in under 8 hours while maintaining up to 95% of full-dataset performance.

Introduction

Image restoration tasks such as deblurring, denoising, and deraining have seen significant progress driven by increasingly large datasets. However, the computational cost of training on these datasets creates a barrier for researchers without access to high-end GPU clusters. While dataset distillation has succeeded in classification, its application to the pixel-level regression required for restoration has been limited. TripleD addresses this gap by focusing on distribution alignment and informative sample selection.

Methodology

The TripleD pipeline consists of three primary stages:

  1. Complexity-Based Selection: Using a pre-trained Vision Transformer (ViT-16), images are ranked by entropy. The top 1–2% most informative samples are selected to form a core real subset.
  2. Latent Distillation (SD3): Instead of pixel-space optimization, TripleD utilizes Stable Diffusion 3 to synthesize high-fidelity training pairs in the latent space, capturing structural information more efficiently.
  3. Distribution Alignment: A lightweight CNN fine-tunes the synthetic data to ensure its feature-level distribution aligns with the original large-scale dataset using L2 loss and KL divergence.
Complexity-based selection ranks images by entropy to identify the most informative samples for distillation.

Key Results and Efficiency

TripleD demonstrates significant efficiency gains across various benchmarks, including deraining, deblurring, and denoising.

  • Resource Savings: Training a 4K resolution restoration model on an NVIDIA RTX 3090 takes less than 8 hours, representing a 500% reduction in compute requirements.
  • Performance Retention: On the Rain100L dataset, TripleD achieves 32.08 dB PSNR using only 2% of the data, compared to 36.52 dB for the full dataset. As training progresses, the distilled dataset captures the vast majority of the performance of the full dataset.
  • Optimal Trade-off: Analysis shows that while performance scales with dataset size, the most significant gains occur within the first 5% of distilled data, with 2% often representing the "Best Trade-off" for practical use.
TripleD achieves a high performance-to-data ratio, with 2% of the dataset providing a "Best Trade-off" for training efficiency.

Discussion and Future Work

TripleD highlights that high-entropy images are more valuable for restoration training. By focusing on the "hard" 2% of the data, the model learns more than it would from a random sample. By distilling into latent space, the framework avoids the noise associated with direct pixel distillation. While a small performance gap remains compared to full-dataset training, TripleD makes state-of-the-art restoration accessible to a wider range of researchers. Future work will explore the relationship between degradation complexity and restoration difficulty to further close the performance gap.

Summary updated

Abstract

With the exponential increase in image data, training an image restoration model is laborious. Dataset distillation is a potential solution to this problem, yet current distillation techniques are a blank canvas in the field of image restoration. To fill this gap, we propose the Distribution-aware Dataset Distillation method (TripleD), a new framework that extends the principles of dataset distillation to image restoration. Specifically, TripleD uses a pre-trained vision Transformer to extract features from images for complexity evaluation, and the subset (the number of samples is much smaller than the original training set) is selected based on complexity. The selected subset is then fed through a lightweight CNN that fine-tunes the image distribution to align with the distribution of the original dataset at the feature level. To efficiently condense knowledge, the training is divided into two stages. Early stages focus on simpler, low-complexity samples to build foundational knowledge, while later stages select more complex and uncertain samples as the model matures. Our method achieves promising performance on multiple image restoration tasks, including multi-task image restoration, all-in-one image restoration, and ultra-high-definition image restoration tasks. Note that we can train a state-of-the-art image restoration model on an ultra-high-definition (4K resolution) dataset using only one consumer-grade GPU in less than 8 hours (500 savings in computing resources and immeasurable training time).
BibTeX
@misc{zheng2025distribution,
  author = {Zhuoran Zheng and Xin Su and Chen Wu and Xiuyi Jia},
  title = {{Distribution-aware Dataset Distillation for Efficient Image Restoration}},
  year = {2025},
  eprint = {2504.14826},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.14826},
}
Figure