Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

Cemil-Andrei Dilmac, Florinel-Alin Croitoru, Radu Tudor IonescuarXiv 2026 · arXiv

TL;DR

When training compact student models from massive datasets using few-shot knowledge distillation, complex data-pruning algorithms mysteriously struggle to beat simple random sampling. This paper introduces few-medoids, an elegantly simple strategy that selects the most geometrically central samples within a teacher’s latent space. Despite its simplicity, few-medoids consistently outperforms random selection and established geometric baselines across diverse image classification datasets and architectures.

What Problem Does This Paper Address?

Modern deep learning relies on massive datasets and over-parameterized models, incurring massive computational and environmental costs. To mitigate this, researchers use two complementary techniques: Knowledge Distillation (KD) (compressing a large "teacher" model into a smaller "student" model) and Coreset Selection (finding a tiny, highly representative subset of data for efficient training).

Intuitively, combining these two approaches for few-shot knowledge distillation—where the student is trained on just a few samples per class using the teacher's guidance—should be a slam dunk. However, a strange paradox exists in current literature: sophisticated coreset selection methods often fail to beat a simple baseline of picking random samples. This happens because traditional methods prioritize highly diverse or "hard" boundary examples, which are often too complex for a small student model to effectively learn from the teacher.

Methods

The authors propose few-medoids, a training-free, geometry-based coreset selection method that leverages the pre-trained teacher's latent space.

The algorithm proceeds class by class:

  1. Feature Extraction: Pass all training images of a given class through the pre-trained teacher model to extract their latent feature representations.
  2. Centrality Scoring: For each sample, compute its average Euclidean ($L_2$) distance to all other samples in the same class.
  3. Ranking & Selection: Sort the samples in ascending order based on this score. The top-ranked sample is the true "class medoid" (the most central point). Select the top $k$ samples to form the coreset.

The student model is then trained on this compact coreset using a standard KD loss function, which balances matching the teacher's soft probability distributions with predicting the ground-truth hard labels.

The authors compare few-medoids against four baselines: random selection, herding (approximating the dataset mean), k-center Greedy (maximizing diversity), and a custom PCA-guided matching baseline.

Datasets

The method was evaluated across four standard image classification benchmarks representing different levels of difficulty and domain types:

  • CIFAR-10 & CIFAR-100: General object classification (32x32 pixels).
  • Oxford Flowers 102: Fine-grained flower species recognition with variable resolutions.
  • Food-101: Challenging food dish classification with variable resolutions.

Detailed dataset statistics can be seen below:

Table 1: Statistics across object class recognition datasets. We report the number of classes, as well as the number of images and the number of images per class for each split: training, validation and test.

Key Insights and Ideas

  • Centrality Over Complexity: In knowledge distillation, "medium-difficulty" or central samples provide a smoother, more stable decision boundary for the student model than hard edge cases. Few-medoids naturally captures these ideal training examples.
  • Latent Space Alignment: By utilizing the teacher's pre-trained latent space to calculate distances, few-medoids inherits semantic relationships that raw pixel spaces cannot capture.
  • Impact of Student Initialization: Latent space analysis reveals that few-medoids helps students trained from scratch form much tighter, more discriminative class clusters than random selection.

Key Results

The authors evaluated three teacher-student pairs spanning both convolutional and transformer architectures (ResNet-34 $\rightarrow$ ResNet-18, ViT-B/16 $\rightarrow$ ResNet-50, and ViT-B/16 $\rightarrow$ ViT-Small) across various coreset sizes ($k \in \{1, 2, 4, 8, 16, 32, 64, 128\}$ samples per class).

  • CIFAR-10: For the ResNet-34 $\rightarrow$ ResNet-18 pair, few-medoids achieved the top accuracy in 5 out of 8 few-shot scenarios, outperforming herding and random selection by over 5% in certain regimes. For the cross-architecture ViT-B/16 $\rightarrow$ ResNet-50 pair, few-medoids dominated for all $k \ge 16$, beating other baselines by up to 11%.
Table 3: Few-shot KD test accuracy (%) on CIFAR-10, for k ∈{1, 2, 4, 8, 16, 32, 64, 128}. Average accuracy rates and standard deviations are computed over five runs with different seeds. Bold green indicates the best result among coreset selection methods for a given k and a given neural model pair, while blue underline indicates the second-best result.
  • CIFAR-100: Few-medoids dominated the low- and mid-$k$ regimes ($k \le 32$), outperforming herding and random selection by up to 4%. However, at larger coreset sizes ($k \ge 64$), herding and random selection began to catch up and overtake it.
Table 4: Few-shot KD test accuracy (%) on CIFAR-100, for k ∈{1, 2, 4, 8, 16, 32, 64, 128}. Average accuracy rates and standard deviations are computed over five runs with different seeds. Bold green indicates the best result among coreset selection methods for a given k and a given neural model pair, while blue underline indicates the second-best result.
  • Food-101: Few-medoids completely dominated, achieving the highest accuracy across all coreset dimensions for the first two model pairs, outperforming herding by up to 8% and random selection by up to 10% at $k=16$.

Limitations and Future Work

While highly effective, few-medoids exhibits a few limitations:

  • Sensitivity to Small Datasets: On Oxford Flowers 102, where each class only contains 10 training images, performance rankings became unstable, likely due to the lack of sufficient data to accurately calculate true class medoids.
  • Pre-trained Student Diminishing Returns: In the ViT-B/16 $\rightarrow$ ViT-Small setup—where the student model is already initialized with pre-trained weights rather than trained from scratch—herding generally prevailed as $k$ increased.

The authors suggest that future work should continue to explore the intersection of data pruning and knowledge distillation to refine how models select optimal training subsets.

Summary updated

Abstract

Coreset selection aims to identify a small and highly representative subset of a massive dataset for efficient model training. The problem remains challenging even in the few-shot knowledge distillation (KD) setup, where a full-scale pre-trained teacher informs the student network. Typical sample selection strategies often struggle to surpass the random selection baseline. In this paper, we showcase few-medoids, an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid (average image) of each class. We present extensive KD experiments on four datasets, covering a wide range of image classification problems, and three teacher-student model pairs, comprising both convolutional and transformer networks. Although the proposed method is embarrassingly simple, our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline, as well as the other coreset selection strategies. We therefore consider that few-medoids can be used as a drop-in replacement for commonly-used baselines (e.g. herding or k-center Greedy), in future research on coreset selection. To reproduce the reported results, we publicly release our code at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.
BibTeX
@misc{dilmac2026few,
  author = {Cemil-Andrei Dilmac and Florinel-Alin Croitoru and Radu Tudor Ionescu},
  title = {{Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation}},
  year = {2026},
  eprint = {2607.05891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.05891},
}
Figure