TL;DR
When training compact student models from massive datasets using few-shot knowledge distillation, complex data-pruning algorithms mysteriously struggle to beat simple random sampling. This paper introduces few-medoids, an elegantly simple strategy that selects the most geometrically central samples within a teacher’s latent space. Despite its simplicity, few-medoids consistently outperforms random selection and established geometric baselines across diverse image classification datasets and architectures.
What Problem Does This Paper Address?
Modern deep learning relies on massive datasets and over-parameterized models, incurring massive computational and environmental costs. To mitigate this, researchers use two complementary techniques: Knowledge Distillation (KD) (compressing a large "teacher" model into a smaller "student" model) and Coreset Selection (finding a tiny, highly representative subset of data for efficient training).
Intuitively, combining these two approaches for few-shot knowledge distillation—where the student is trained on just a few samples per class using the teacher's guidance—should be a slam dunk. However, a strange paradox exists in current literature: sophisticated coreset selection methods often fail to beat a simple baseline of picking random samples. This happens because traditional methods prioritize highly diverse or "hard" boundary examples, which are often too complex for a small student model to effectively learn from the teacher.
Methods
The authors propose few-medoids, a training-free, geometry-based coreset selection method that leverages the pre-trained teacher's latent space.
The algorithm proceeds class by class:
- Feature Extraction: Pass all training images of a given class through the pre-trained teacher model to extract their latent feature representations.
- Centrality Scoring: For each sample, compute its average Euclidean ($L_2$) distance to all other samples in the same class.
- Ranking & Selection: Sort the samples in ascending order based on this score. The top-ranked sample is the true "class medoid" (the most central point). Select the top $k$ samples to form the coreset.
The student model is then trained on this compact coreset using a standard KD loss function, which balances matching the teacher's soft probability distributions with predicting the ground-truth hard labels.
The authors compare few-medoids against four baselines: random selection, herding (approximating the dataset mean), k-center Greedy (maximizing diversity), and a custom PCA-guided matching baseline.
Datasets
The method was evaluated across four standard image classification benchmarks representing different levels of difficulty and domain types:
- CIFAR-10 & CIFAR-100: General object classification (32x32 pixels).
- Oxford Flowers 102: Fine-grained flower species recognition with variable resolutions.
- Food-101: Challenging food dish classification with variable resolutions.
Detailed dataset statistics can be seen below:
Key Insights and Ideas
- Centrality Over Complexity: In knowledge distillation, "medium-difficulty" or central samples provide a smoother, more stable decision boundary for the student model than hard edge cases. Few-medoids naturally captures these ideal training examples.
- Latent Space Alignment: By utilizing the teacher's pre-trained latent space to calculate distances, few-medoids inherits semantic relationships that raw pixel spaces cannot capture.
- Impact of Student Initialization: Latent space analysis reveals that few-medoids helps students trained from scratch form much tighter, more discriminative class clusters than random selection.
Key Results
The authors evaluated three teacher-student pairs spanning both convolutional and transformer architectures (ResNet-34 $\rightarrow$ ResNet-18, ViT-B/16 $\rightarrow$ ResNet-50, and ViT-B/16 $\rightarrow$ ViT-Small) across various coreset sizes ($k \in \{1, 2, 4, 8, 16, 32, 64, 128\}$ samples per class).
- CIFAR-10: For the ResNet-34 $\rightarrow$ ResNet-18 pair, few-medoids achieved the top accuracy in 5 out of 8 few-shot scenarios, outperforming herding and random selection by over 5% in certain regimes. For the cross-architecture ViT-B/16 $\rightarrow$ ResNet-50 pair, few-medoids dominated for all $k \ge 16$, beating other baselines by up to 11%.
- CIFAR-100: Few-medoids dominated the low- and mid-$k$ regimes ($k \le 32$), outperforming herding and random selection by up to 4%. However, at larger coreset sizes ($k \ge 64$), herding and random selection began to catch up and overtake it.
- Food-101: Few-medoids completely dominated, achieving the highest accuracy across all coreset dimensions for the first two model pairs, outperforming herding by up to 8% and random selection by up to 10% at $k=16$.
Limitations and Future Work
While highly effective, few-medoids exhibits a few limitations:
- Sensitivity to Small Datasets: On Oxford Flowers 102, where each class only contains 10 training images, performance rankings became unstable, likely due to the lack of sufficient data to accurately calculate true class medoids.
- Pre-trained Student Diminishing Returns: In the ViT-B/16 $\rightarrow$ ViT-Small setup—where the student model is already initialized with pre-trained weights rather than trained from scratch—herding generally prevailed as $k$ increased.
The authors suggest that future work should continue to explore the intersection of data pruning and knowledge distillation to refine how models select optimal training subsets.
Summary updated