FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui WangarXiv 2026 · arXiv

TL;DR

Existing dataset distillation methods struggle with fine-grained datasets because they rely on coarse class labels, leading to synthetic images that lack distinct local features or become overly repetitive. FD$^2$ (Fine-grained Dataset Distillation) solves this by using counterfactual attention learning to pinpoint localized discriminative regions, enforcing class prototype alignment for better separation, and penalizing redundant attention to ensure diverse synthetic samples. This plug-and-play framework seamlessly integrates into decoupled distillation pipelines, delivering massive accuracy boosts (up to 15%) with minimal computational overhead.

What Problem Does This Paper Address?

Dataset distillation (DD) aims to compress massive training sets into a tiny handful of synthetic images (e.g., 1 or 10 images per class) while preserving the ability to train accurate models. Recently, "decoupled" DD methods have dramatically improved efficiency by breaking the process into three independent stages: model pretraining, sample distillation, and soft-label generation.

However, when applied to fine-grained datasets—where categories share highly similar global structures and differ only in subtle, localized details (like distinguishing a Sage Thrasher from a Brown Creeper)—standard decoupled methods fail. The authors identify two core limitations:

  1. Unfavorable Fine-Grained Characteristics: Coarse class-label supervision fails to enforce intra-class compactness or inter-class separability. As a result, distilled images inherit massive intra-class variance and subtle inter-class boundaries, making it incredibly difficult for a student model to learn.
  2. Within-Class Homogenization: Because synthetic images are optimized sample-by-sample under nearly identical conditions, images within the same class converge toward identical solutions. They end up focusing on the exact same local features, drastically reducing the diversity of visual cues available to the student model.
Fig. 1: (a) On CUB-200-2011, 10 classes are randomly sampled to compare the origi- nal training set and the SRe2L++ distilled set, where intra-class dispersion (bars) and inter-class distance (line) are reported. (b) Attention heatmaps of Black-footed Alba- tross in CUB-200-2011, comparing the attended regions of distilled samples produced by SRe2L++ and FD2.

Methods

To overcome these challenges, the authors introduce FD$^2$, a dedicated framework that injects fine-grained supervision into the decoupled distillation pipeline using three main components:

  1. Counterfactual Attention Learning (CAL): During the teacher pretraining phase, the network learns to extract precise attention maps by looking at factual vs. counterfactual scenarios. If the network perturbs an attention map and the class evidence drops, it knows it has successfully localized a truly discriminative region. This phase also maintains online class prototypes (feature centers) for every category.
  2. Fine-Grained Characteristic Constraint ($\mathcal{L}_F$): During distillation, this constraint acts as a force field in the feature space. It pulls each synthetic image closer to its target-class prototype while pushing it away from other-class prototypes, directly optimizing for intra-class compactness and inter-class separability.
  3. Similarity Constraint ($\mathcal{L}_S$): To prevent images within the same class from becoming identical copies, the similarity constraint penalizes overlapping attention maps within a sequential group of distilled images. This forces the optimization process to explore and cover different discriminative regions (e.g., one image captures the bird's beak, another captures its wing patterns).
Overview of the FD$^2$ framework

Datasets

The framework was rigorously evaluated across both fine-grained and general benchmarks:

  • Fine-Grained Datasets: CUB-200-2011 (birds), FGVC-Aircraft, and Stanford Cars.
  • General Datasets: ImageNette and ImageWoof (a challenging 10-class fine-grained subset of ImageNet).

Key Insights and Ideas

  • Coarse Labels are Not Enough: Relying solely on global cross-entropy loss during distillation ignores the structural geometry of the feature space. Explicitly aligning features with class prototypes is essential when class differences are subtle.
  • Optimization Drives Redundancy: Standard sample-wise iterative synthesis naturally leads to redundant features because the optimization landscape lacks any mechanism to encourage diversity across samples.
  • Attention as a Diversity Metric: By treating attention maps as distributions and maximizing the distance between them for same-class images, you can effectively force the generator to synthesize complementary visual features across the distilled set.

Key Results

FD$^2$ was integrated as a plug-in module into state-of-the-art decoupled distillation methods like SRe$^2$L++ and FADRM+, yielding substantial performance gains across the board, particularly in ultra-low budget settings (Images Per Class, IPC = 1):

  • Stanford Cars (IPC=1): Integrating FD$^2$ into SRe$^2$L++ boosted ResNet18 top-1 accuracy from 52.4% to 64.5% (+12.1%). For FADRM+, accuracy jumped from 60.3% to 74.1% (+13.8%).
  • CUB-200-2011 (IPC=1): SRe$^2$L+++FD$^2$ achieved a +9.0% accuracy improvement using a ResNet50 student.
  • Cross-Architecture Generalization: Datasets distilled using an FD$^2$-augmented ResNet18 teacher successfully transferred to completely different student architectures like ShuffleNetV2, MobileNetV2, DenseNet121, and ConvNeXt-Tiny, consistently outperforming the baselines.
  • Efficiency: Unlike traditional matching-based distillation methods (like DSA or DATM) which demand massive GPU memory (up to 68 GB) and long optimization times, FD$^2$ maintains the lightweight footprint of decoupled methods, adding only a tiny fraction of time and memory overhead (e.g., peak GPU memory only increases from 4.8 GB to 5.3 GB when added to SRe$^2$L++).

Limitations and Future Work

While FD$^2$ excels on fine-grained datasets and higher-capacity student models, the authors noted that its performance gains are more limited when training small-capacity students (like ResNet18) on simpler, general-purpose datasets (like ImageNette).

Moving forward, the authors plan to optimize FD$^2$ for broader general datasets, adapt the framework to Vision Transformer (ViT) architectures, and scale the pipeline up to handle massive, large-scale datasets like ImageNet-1K.

Summary updated

Abstract

Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks. Decoupled DD further improves efficiency by splitting the pipeline into pretraining, sample distillation, and soft-label generation. However, existing decoupled methods largely rely on coarse class-label supervision and optimize samples within each class in a nearly identical manner. On fine-grained datasets, this often yields distilled samples that (i) retain large intra-class variation with subtle inter-class differences and (ii) become overly similar within the same class, limiting localized discriminative cues and hurting recognition. To solve the above-mentioned problems, we propose FD$^{2}$, a dedicated framework for Fine-grained Dataset Distillation. FD$^{2}$ localizes discriminative regions and constructs fine-grained representations for distillation. During pretraining, counterfactual attention learning aggregates discriminative representations to update class prototypes. During distillation, a fine-grained characteristic constraint aligns each sample with its class prototype while repelling others, and a similarity constraint diversifies attention across same-class samples. Experiments on multiple fine-grained and general datasets show that FD$^{2}$ integrates seamlessly with decoupled DD and improves performance in most settings, indicating strong transferability. Code is available at https://github.com/Guang000/FD2.
BibTeX
@misc{ma2026fd,
  author = {Hongxu Ma and Guang Li and Shijie Wang and Dongzhan Zhou and Baoli Sun and Takahiro Ogawa and Miki Haseyama and Zhihui Wang},
  title = {{FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation}},
  year = {2026},
  eprint = {2603.25144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.25144},
}
Figure