TL;DR
DeCO (Discriminative Evidence Composition) reimagines dataset distillation for fine-grained visual classification by shifting the focus from global image statistics to localized, class-specific cues. By mining informative patches using transformer attention, enforcing spatial diversity, and packing these patches into grid-composed images, DeCO achieves massive accuracy gains under tight data budgets (like 1 image per class) using standard hard-label training.
What Problem Does This Paper Address?
Dataset distillation (DD) aims to compress massive training datasets into a handful of synthetic images while retaining as much downstream model performance as possible. While existing DD methods excel at general image classification, they struggle with Fine-Grained Visual Classification (FGVC)—distinguishing between highly similar categories like specific bird species, aircraft models, or car trims.
In FGVC, the magic lies in the details: a subtle beak shape, a specific wing texture, or a unique headlight design. Standard distillation methods optimize for global image statistics, which often wastes precious pixel budgets on background noise or redundant features. When you only have a budget of 1 to 5 images per class (Images Per Class, or IPC), filling half the image with sky or grass is a massive missed opportunity.
Methods
The authors propose Discriminative Evidence Composition (DeCO), a framework that treats fine-grained dataset distillation as a problem of budgeted, diverse evidence preservation.
The pipeline consists of three main stages:
- Discriminative Evidence Mining: DeCO uses a pretrained fine-grained model (a TransFG Vision Transformer) as a teacher. By applying attention rollout, it scores image patches based on how much the model relies on them to make a classification.
- Spatial Suppression: To prevent the system from repeatedly selecting the exact same high-scoring area (e.g., picking four patches from the same bird wing), DeCO applies a greedy distance-based constraint. A patch is only kept if it is sufficiently far from previously selected patches, forcing the algorithm to find complementary features (like the head, tail, and legs).
- Evidence Bank & Grid Composition: The selected patches are cropped and stored in class-wise "evidence banks." To create a distilled image, DeCO samples $M = g^2$ patches (typically a $2 \times 2$ grid) from the same class bank and stitches them together.
Crucially, the teacher model is completely discarded after dataset creation. Downstream student models are trained on these composite grid images from scratch using standard hard labels, without needing soft labels or teacher logits.
Datasets
The authors evaluated DeCO on three widely recognized fine-grained visual classification benchmarks:
- CUB-200-2011: 200 bird species.
- FGVC-Aircraft: 100 aircraft variants.
- Stanford Cars: 196 car categories.
All evaluations were conducted under strict data constraints with IPC budgets of 1, 3, and 5.
Key Insights and Ideas
- Maximizing Pixel Utility: Instead of forcing a model to learn from a complete image where the background dominates, grid composition ensures that every single quadrant of the synthetic image contains high-density, class-relevant information.
- The Power of Spatial Diversity: The paper demonstrates that simply picking the highest-attention patches isn't enough. Enforcing spatial suppression acts as an automated way to capture different anatomical parts of an object, which is exactly how humans perform fine-grained recognition.
- Hard Labels are Enough: A common trend in dataset distillation is relying on complex soft-label matching or keeping the teacher in the loop during student training. DeCO proves that if the visual evidence in the distilled images is clean and concentrated enough, standard hard-label training is entirely sufficient.
Key Results
DeCO achieved substantial performance leaps over representative coreset and dataset-distillation baselines across all datasets and IPC budgets.
The most dramatic improvements occurred at IPC = 1:
- CUB-200-2011: DeCO achieved 65.53% top-1 accuracy, outperforming the patch-based baseline RDED (38.25%) by over 27 percentage points.
- FGVC-Aircraft: DeCO reached 66.04% (vs. RDED's 22.11%).
- Stanford Cars: DeCO hit 63.94% (vs. RDED's 16.95%).
Furthermore, DeCO consistently outperformed a variant called $\text{DeCO}_{\text{rand}}$ (which uses random patch cropping instead of attention guidance), proving that targeted, discriminative region selection is vital to the method's success.
Limitations and Future Work
According to the paper's checklist, the authors note a few limitations:
- Pretrained Teacher Dependence: DeCO relies heavily on a high-quality, pretrained fine-grained teacher model to bootstrap the attention maps.
- Upfront Computational Cost: The construction stage involves running attention rollout and spatial suppression across the dataset, which incurs an upfront computational overhead.
- Scope of Evaluation: The method was specifically tailored for and evaluated on fine-grained benchmarks; its behavior on broader, more heterogeneous datasets (like ImageNet) remains an open question for future exploration.
Summary updated