TL;DR
SilverSpoon-KD is a new open-source PyTorch toolkit that unifies the fragmented world of Knowledge Distillation (KD) into a single, scalable API. It introduces the first general-purpose implementation of blockwise distillation and consistently reduces GPU memory usage by 13–24% compared to existing specialized tools, enabling researchers to distill 14B-parameter models on modest hardware.
What Problem Does This Paper Address?
Knowledge Distillation—the process of transferring "knowledge" from a large teacher model to a smaller student model—is a cornerstone of modern model deployment. However, the tooling has failed to keep up with research.
As shown in Figure 1, KD is a combinatorial explosion of workloads (Vision, LLMs), architectural relationships (homogeneous vs. heterogeneous), and paradigms (matching outputs vs. matching internal features). Most existing toolkits only support "narrow slices" of this space. If you want to perform blockwise distillation (training student blocks in isolation) on a transformer, you are often forced to write a bespoke pipeline from scratch. Furthermore, many toolkits lack the engineering "plumbing" required for modern scale, such as integration with Hugging Face’s Trainer or FSDP sharding.
Methods
The authors present SilverSpoon-KD, built on top of the Hugging Face Trainer to inherit standard deep learning boilerplate while adding KD-specific logic. Its core design rests on three pillars:
- Unified API: Switching between paradigms—like Response-based (ResKD), Holistic (HKD), or Blockwise (BKD)—is as simple as a class swap in Python.
- Flexible Alignments: A pairing API allows users to map teacher modules to student modules at any granularity (whole layers or specific sub-modules) using simple regex or lists.
- Scalable Engineering:
- Auto-projectors: Automatically detects and bridges hidden-dimension mismatches between teacher and student.
- Parallelism: Uses independent CUDA streams to overlap teacher and student forward passes, and supports FSDP sharding to split large teacher models across multiple GPUs.
- Blockwise KD (BKD): SilverSpoon-KD provides a general abstraction for BKD, where student blocks are trained against teacher-forced inputs. This decouples gradients, significantly lowering peak memory.
Datasets
The toolkit was validated across eight diverse experiments using:
- Vision: CIFAR-10 and CIFAR-100 (VGG and ResNet architectures).
- Language (Encoder): Dolma (pre-training) and MNLI (downstream tasks) for BERT-style models.
- Language (Causal/LLM): Dolma, LAMBADA, HellaSwag, and MMLU for GPT-2 and Qwen3 models.
Key Insights and Ideas
- Multi-stage Chaining is Essential: One of the paper's strongest findings is that the best results almost always come from "chaining" paradigms (e.g., starting with Blockwise KD, moving to Holistic KD, and finishing with a LoRA fine-tune). SilverSpoon-KD makes this composition trivial.
- The Paradigm-Compression Interaction: The unified API allowed the authors to discover that while Blockwise KD is excellent for mild compression, it can become "actively harmful" during aggressive compression where the structural gap between teacher and student is too wide.
- Zero-Overhead Distillation: Because Blockwise KD trains blocks independently, it can provide distillation benefits at the same GPU memory cost as standard training. This is a "unique practical advantage" for researchers constrained by hardware.
Key Results
The authors benchmarked SilverSpoon-KD against four major peers (torchdistill, TextBrewer, torchtune, and DistillKit).
- Memory Efficiency: SilverSpoon-KD won on peak memory in all five head-to-head settings, with reductions ranging from 13% to 24.4%.
- Throughput: Performance was on par with or better than specialized toolkits, despite SilverSpoon-KD having a much broader feature set.
- Breaking the Capability Frontier: In a stress test with a 14B-parameter teacher on two 40GB A100 GPUs, SilverSpoon-KD was the only toolkit able to scale beyond a batch size of 1. Other toolkits either crashed with Out-of-Memory (OOM) errors or were forced to use inefficiently small batches because they couldn't shard the teacher model.
Limitations and Future Work
The authors acknowledge a few current boundaries:
- White-box Assumption: The toolkit requires "white-box" access to the teacher's internal states and logits, ruling out "black-box" distillation from proprietary APIs.
- Single Student/Teacher: Currently, the system is designed for one student learning from one teacher; multi-teacher distillation is a planned extension.
- Configuration: While the programmatic Python API is flexible, the authors suggest adding a declarative (YAML/JSON) layer for users who prefer configuration files over code.
SilverSpoon-KD is open-source and available on GitHub.
Summary updated