SAS: Semantic-aware Sampling for Generative Dataset Distillation

Mingzhuo Li, Guang Li, Linfeng Ye, Jiafeng Mao, Takahiro Ogawa, Konstantinos N. Plataniotis, Miki HaseyamaarXiv 2026 · arXiv

TL;DR

Dataset distillation compresses massive training datasets into tiny, high-quality synthetic subsets to drastically speed up neural network training. This paper introduces Semantic-aware Sampling (SAS), a post-sampling framework that uses pretrained CLIP embeddings to select the most semantically distinct and diverse images from a generated image pool. By filtering out ambiguous and redundant samples, SAS significantly boosts downstream model accuracy across multiple benchmarks with negligible computational overhead.

What Problem Does This Paper Address?

Training state-of-the-art deep neural networks requires massive datasets (like ImageNet), which demand immense storage and days of computation. Dataset Distillation (DD) solves this by synthesizing a tiny fraction of data—often just 10 to 50 images per class (IPC)—that can train a model to nearly the same performance as the full dataset.

Recently, generative dataset distillation (using diffusion models like Diffusion Transformers, or DiT) has become the dominant paradigm. These methods generate large pools of synthetic images by matching the statistical distributions or training trajectories of the original dataset. However, they suffer from a major flaw: they ignore high-level semantics. As a result, the generated image pools often contain semantically ambiguous, noisy, or highly redundant images that waste the precious, limited slots available in a distilled dataset.

Methods

The authors propose Semantic-aware Sampling (SAS), a plug-and-play post-sampling framework that acts as an intelligent filter on top of existing generative distillation methods.

Instead of relying purely on statistical matching, SAS leverages a pretrained CLIP model as a semantic prior. It maps the generated image pool and class text prompts into a shared embedding space and evaluates each image using three novel scoring functions based on angular distance:

  1. Target Relevance Score ($s_{rel}$): Measures how well an image aligns with its intended class prompt (e.g., "A photo of a Samoyed").
  2. Non-target Separation Score ($s_{sep}$): Measures how far the image is from the most confusing alternative class, ensuring clear decision boundaries.
  3. Intra-set Diversity Score ($s_{div}$): Measures the average distance between an image and all other selected images of the same class to prevent near-duplicates.
FIGURE 2. Calculation of semantic scores. The image pool is constructed using existing generative dataset distillation models and encoded with a pre-trained CLIP model, along with text prompts of the corresponding classes. Three semantic scores are calculated in the semantic space using angular distance.

SAS implements these scores via a two-stage adaptive sampling strategy:

  • Stage 1 (Score Filtering): Combines relevance and separation into a single margin score to filter out ambiguous or noisy images, keeping the top 50% as a reliable candidate set.
  • Stage 2 (Diversity-aware Selection): Dynamically builds the final distilled dataset. Instead of using a static diversity score, it greedily evaluates and updates the diversity score relative to the currently selected subset, effectively maximizing semantic coverage.

Datasets

The authors evaluated SAS on several widely used image classification benchmarks:

  • ImageWoof: A challenging 10-class subset of ImageNet consisting of look-alike dog breeds, ideal for testing fine-grained classification.
  • ImageNette: An easier 10-class subset of ImageNet with distinct objects.
  • ImageIDC: A randomly selected 10-class subset of ImageNet representing a more general data pattern.
  • CIFAR-10: A standard dataset used to verify generalization across different resolutions and visual domains.

Key Insights and Ideas

  • Statistical Alignment $\neq$ Semantic Clarity: Generative models can perfectly trick distribution-matching algorithms while still producing images that are visually confusing or redundant. High-level semantic guidance is necessary to curate the best training samples.
  • Dynamic Diversity over Static Ranking: You cannot judge if an image is redundant based on the whole pool; you must judge it against what you have already chosen. Evaluating diversity dynamically as the subset evolves yields far better data coverage.
  • Angular Distance Beats Cosine Similarity: In high-similarity regions of CLIP space, standard cosine similarity tends to compress numerical variations. Angular distance preserves these subtle relative differences, allowing for finer discrimination.

Key Results

SAS consistently outperformed baseline generative methods (like DiT and Minimax) across all datasets, architectures (ConvNet, ResNet-18), and IPC settings:

  • Accuracy Boost: On the challenging ImageWoof dataset at 50 IPC, applying SAS to a Minimax-generated pool boosted ResNet accuracy from 54.4% to 59.9% (a +5.5% absolute improvement).
  • Minimal Overhead: Generating a synthetic image pool takes roughly 2,100 seconds on an NVIDIA RTX A6000. SAS completes its two-stage scoring and sampling process in just 23 seconds, adding negligible overhead for a massive performance gain.
FIGURE 3. t-SNE analysis on the dataset sampled by SAS from the image pool. Overall, sampled datasets are distributed diversely inside the class cluster while maintaining relevance to the class center. The analysis is conducted on ImageWoof with IPC of 10.

As shown in the t-SNE analysis above, the final datasets sampled by SAS are distributed diversely within their respective class clusters while maintaining strong relevance to the class centers.

Limitations and Future Work

The primary limitation of SAS is its heavy reliance on a pretrained CLIP model. In highly specialized domains where CLIP lacks reliable semantic alignment—such as medical imaging (e.g., X-rays) or satellite remote sensing—the proposed scoring functions may become less informative.

The authors note that future work will explore integrating domain-adaptive semantic encoders to handle out-of-distribution domains and extending the SAS framework to downstream tasks beyond image classification, such as object detection or segmentation.

Summary updated

Abstract

Deep neural networks have achieved impressive performance across a wide range of tasks, but this success often comes with substantial computational and storage costs due to large-scale training data. Dataset distillation addresses this challenge by constructing compact yet informative datasets that enable efficient model training while maintaining downstream performance. However, most existing approaches primarily emphasize matching data distributions or downstream training statistics, with limited attention to preserving high-level semantic information in the distilled data. In this work, we introduce a semantic-aware perspective for dataset distillation by leveraging Contrastive Language-Image Pretraining (CLIP) as a semantic prior for post-sampling. Our goal is to obtain distilled datasets that are not only compact but also semantically class-discriminative and diverse. To this end, we design three semantic scoring functions that quantify class relevance, inter-class separability, and intra-set diversity in a pretrained semantic space. Based on image pools generated by existing distillation methods, we further develop a two-stage strategy for effective sampling: the first stage filters semantically discriminative samples to form a reliable candidate set, and the second stage performs a dynamic diversity-aware selection to reduce redundancy while preserving semantic coverage. Extensive experiments across multiple datasets, image pools, and downstream models demonstrate consistent performance gains, highlighting the effectiveness of incorporating semantic information into dataset distillation.
BibTeX
@misc{li2026sas,
  author = {Mingzhuo Li and Guang Li and Linfeng Ye and Jiafeng Mao and Takahiro Ogawa and Konstantinos N. Plataniotis and Miki Haseyama},
  title = {{SAS: Semantic-aware Sampling for Generative Dataset Distillation}},
  year = {2026},
  eprint = {2605.18012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18012},
}
Figure