When a small "student" network is trained to imitate a large "teacher", the common practice is to make the student's hidden activations match the teacher's. In the teacher's last layers this matching can backfire. Those layers mix many signals across their channels, and a small student cannot reproduce that mixture. On ImageNet, matching a ViT-Large teacher's last-layer features gives a DeiT-Tiny student lower accuracy than matching output probabilities alone (Distillation Dynamics). Interpretability research reports the reverse finding for the same layers. Non-negative matrix factorization (NMF) writes a matrix of non-negative activations as a purely additive combination of a few non-negative basis vectors. Applied to a network's penultimate layer, it recovers a small set of stable, human-readable "parts", such as wings, ears or eyes. A classifier fed the reconstruction from about 30 of these parts loses only a few points of accuracy (ICE; A Holistic Approach to Unifying Automatic Concept Extraction).
This proposal puts the two findings together. We factorize the teacher's late-layer activations once, offline, into a fixed dictionary of non-negative parts. We then train the student to predict how much of each part is present at each location, weighting each part by how much the teacher's decision depends on it. The student does not try to reproduce the teacher's full activation tensor. It reproduces a short list of additive part strengths that the teacher's classifier actually uses. The claim is that this removes the negative transfer of late-layer distillation and gives a student whose internal parts can be inspected. The method is built to fail visibly if the gain comes only from low-rank compression and not from non-negativity.
Core Claim
Late-layer feature distillation fails because it transfers everything, including directions that are expensive to reproduce and that the teacher's classifier barely uses. The problem is not the late-layer information itself. NMF with importance weighting separates the two cases. Its additive parts are the units the teacher's classifier reads, because a reconstruction from them preserves accuracy (ICE). Low-importance parts can be down-weighted or dropped. The specific, testable prediction is this: on the DeiT-Tiny-from-ViT-Large setting where last-layer matching loses to output matching, importance-weighted NMF part distillation should beat output matching. A PCA dictionary of the same rank, trained the same way, should not. If PCA does just as well, non-negativity is not the active ingredient, and we will report that.
Relation to Prior Work
Two groups of papers motivate the idea. Distillation work shows that matching intermediate layers is often the most important source of distillation gains (Distiller). It also shows that matching late layers in transformers can hurt (Distillation Dynamics). The second paper is the negative result our method must overturn.
A second group supplies the method. ICE factorizes post-ReLU CNN feature maps with NMF and shows the factorization can be inverted: the original activations can be rebuilt from the parts and fed back into the classifier. A Holistic Approach to Unifying Automatic Concept Extraction argues that the penultimate layer gives the most faithful parts and pairs extraction with an estimate of each part's importance. Less is More compresses thousands of attribution maps into about ten NMF components. All three study a frozen network after training. We reuse their extraction step and their importance estimates, but as training targets for a different network. We use classic multiplicative-update NMF (Algorithms for Non-negative Matrix Factorization) to fit the dictionary once.
The closest existing combinations differ in mechanism. Concept Distillation uses a DINO teacher's concept directions to guide a student. Those directions are learned by linear classifiers trained on 50 to 150 curated example images per concept, are not constrained to be non-negative, and are transferred through a directional-derivative loss aimed at debiasing. The authors name automatic concept discovery as open work. Our parts are discovered without labels and the goal is compression. FedSPLIT pairs NMF with a step it calls distillation, but in federated recommendation: an algebraic update of rating-matrix factors with no neural layer matching. It shares the vocabulary but not the problem.
Method
Let $a_i \in \mathbb{R}_+^{n}$ be the teacher's late-layer activation at spatial position or token $i$, with $n$ the channel width. We fit a dictionary $V \in \mathbb{R}_+^{n \times k}$ of $k$ parts on activations from a subset of training images, and keep it frozen. For each training image, the teacher's part strengths $u_i \in \mathbb{R}_+^{k}$ are the non-negative least-squares solution of $a_i \approx V u_i$. These can be computed in a few projected-gradient steps because $V$ is fixed, or cached ahead of time. A lightweight head on the student (a 1×1 convolution or linear map $P$ followed by a ReLU) produces predicted strengths $\hat u_i = \mathrm{ReLU}(P s_i)$ from the student activation $s_i$. The parts loss is
$$ \mathcal{L}_{\text{parts}} = \sum_i \sum_{c=1}^{k} w_c \left(\hat u_{ic} - \bar u_{ic}\right)^2 , $$
where $\bar u_{ic}$ is $u_{ic}$ normalized by that part's average strength, so that parts with large activations do not dominate. The weight $w_c$ is part $c$'s importance: the average drop in the teacher's true-class score when part $c$ is removed from the reconstruction $V u$ and the result is fed into the teacher's classifier head. This uses the invertibility shown in ICE. The student's total loss is cross-entropy, plus a standard output-matching term, plus $\lambda \mathcal{L}_{\text{parts}}$.
There is one real technical obstacle. CNN post-ReLU activations are non-negative, but ViT token features are not. For transformers we take the post-GELU hidden activations inside the last MLP block, which are nearly non-negative, and clip them at zero. As a check, we also try splitting the residual-stream features into positive and negative halves, $[x]_+$ and $[-x]_+$, and factorizing the concatenation. Which of these keeps accuracy under the ICE reconstruction test decides which one we use. That test is run before any distillation.
How to choose the number of parts $k$ is an open problem in this area (Few-shot learning in medical imaging). We do not hide it. We sweep $k$ over 16, 32, 64 and 128. We also report a cheap selection rule, picking the smallest $k$ at which the teacher's reconstruction-fed accuracy is within 2% of the original, and test whether that rule predicts the best $k$ for the student.
Research Program
The work has three stages. The first is a diagnostic. For the ViT-Large teacher and the ResNet teachers, we measure accuracy when reconstructions from $k$ parts are fed into the teacher's own head, using both NMF and PCA dictionaries. This tells us whether the late-layer signal the classifier needs fits in a few additive parts. The second stage is distillation under matched budgets on ImageNet-1K. The main setting is DeiT-Tiny from ViT-Large, chosen because last-layer matching is documented to fail there. A secondary setting is a ResNet-18 student, the setting in which DKD's ImageNet result was reported (Decoupled Knowledge Distillation); we use the teacher from that setup. The third stage tests the cross-architecture case, a CNN teacher with a ViT student. Here raw feature matching runs into incompatible spatial layouts and activation distributions (Kernel Based Progressive Distillation). Part strengths are pooled to the student's token grid, which removes the dependence on channel coordinates. Throughout, we check on CUB-200-2011 whether the student's part maps fall on the same image regions as the teacher's. We use the expert-annotated subset used by Less is More and measure overlap (IoU) with the expert masks.
Baselines
The naive reference is the student trained on labels alone, together with plain output distillation (SoftKD). In the main ViT setting, Distillation Dynamics reports 76.85% top-1 for DeiT-Tiny from ViT-Large with SoftKD. That is the number we must beat. The same paper reports 76.58% for last-layer SpectralKD, which is the failure we aim to reverse. We rerun both with our training recipe instead of quoting them, so every comparison uses the same epochs, augmentation and optimizer.
The strongest output-only competitor is Decoupled Knowledge Distillation. DKD separates the loss on the target class from the loss on the non-target classes. It reports 71.70% top-1 for a ResNet-18 student on ImageNet-1K, against 70.66% for classic distillation. It is the strongest competitor because it adds no parameters and is already competitive with feature-based methods. In the ResNet-18 setting we use it as the output-matching term inside our own loss, so any gain over 71.70% comes from the parts loss on top of DKD. We give DKD's two weights the same tuning budget we give $\lambda$ and $k$.
For feature methods, we compare against plain L2 regression through a learned projector on the same late layer, cosine-similarity matching (CSKD), and relation-based matching of sample-pair kernels (Feature Kernel Distillation; Relational Knowledge Distillation). Each gets the same layer choice and the same number of tuning trials.
The ablations that isolate our key ingredient are four. The first swaps NMF for PCA at the same rank, with the same projection head; the ReLU is removed because PCA coefficients are signed. The second uses NMF parts with uniform weights instead of importance weights. The third uses a random non-negative dictionary of the same rank. The fourth uses NMF reconstruction matching, $\| V\hat u - a\|^2$, instead of coefficient matching.
To count as beating the baselines, part distillation must exceed SoftKD's 76.85% in the ViT setting by more than the seed-to-seed spread over three seeds. It must also beat the PCA ablation by the same margin. In the ResNet-18 setting it must improve on DKD alone. For the cross-architecture and CUB experiments, the file has no reported numbers: no reported result found on Lacuna for CNN-to-ViT feature distillation on ImageNet, and no numeric IoU for DCNE on the CUB expert subset. There we compare only against our own reruns.
Why not just…?
Why not just use DKD and skip features entirely? In the ViT setting this is a real threat, because output matching is exactly the baseline that late-layer matching currently loses to. Our answer is empirical. If part distillation plus output matching does not beat output matching alone, the method has no accuracy case, and only the interpretability claim on CUB remains. Two things outputs cannot supply. Output distillation needs a shared label space, while feature-level transfer still works across datasets, as with the +2.75% from CIFAR-100 to STL-10 in Feature Kernel Distillation. Outputs also carry no information about where in the image the evidence is.
Why not just match a PCA projection of the teacher's features? This is the most dangerous alternative. It might work for the same reason as our method: a low-rank target filters out directions that are expensive to reproduce. Prior evidence favours NMF over PCA only for interpretability and stability (A Holistic Approach to Unifying Automatic Concept Extraction) and for few-shot subspaces (Few-shot learning in medical imaging). Nobody has tested it for distillation. The rank-matched PCA ablation settles the question. If PCA matches NMF in accuracy, we report that low rank alone explains the gain, and NMF's remaining benefit is that the student's parts can be inspected, which we measure with CUB IoU.
Why not just use relation-based distillation, which already avoids coordinate mismatch? Matching the pairwise similarities between samples does avoid aligning channels. But it transfers the teacher's whole similarity structure, including the parts of that structure its classifier does not use, so it does not address the problem that importance weighting targets. It is a direct baseline in both the same-architecture and cross-architecture settings. If it equals our method in the cross-architecture case, we drop that claim.
Why not just distill earlier layers, where matching is not reported to hurt? That avoids the failure rather than explaining it, and early layers carry less of the class-specific part structure. We include mid-layer L2 matching as a baseline. We also test whether combining mid-layer matching with late-layer parts adds anything beyond either one alone.
Decisive Checks
The project turns on three results. First, the teacher-reconstruction diagnostic must show that a moderate $k$ keeps the ViT-Large teacher's late-layer accuracy. If it does not, the targets throw away what the student needs, and we stop before distillation. Second, in the DeiT-Tiny setting, part distillation must beat our rerun of SoftKD (reported at 76.85%), and importance-weighted NMF must beat rank-matched PCA. Third, on CUB, the student's part maps must overlap the expert masks more than maps from an L2-distilled student, measured by the same projection onto the teacher's dictionary. A positive result on the second check alone would show that late-layer negative transfer comes from transferring too much, not from the late layer itself. A positive result on the PCA comparison would show that additivity is what makes the reduced target learnable.
References
- Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers
- Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors
- A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation
- Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
- Less is More: Discovering Concise Network Explanations
- Algorithms for Non-negative Matrix Factorization
- Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement
- FedSPLIT: One-Shot Federated Recommendation System Based on Non-negative Joint Matrix Factorization and Knowledge Distillation
- Few-shot learning for inference in medical imaging with subspace feature representations
- Decoupled Knowledge Distillation
- Kernel Based Progressive Distillation for Adder Neural Networks
- Cosine similarity-guided knowledge distillation for robust object detectors
- Feature Kernel Distillation
- Relational Knowledge Distillation