Distilling What Separates Classes: Fisher-Weighted Feature Distillation

When a large "teacher" network is compressed into a small "student", the most effective methods do not only match the teacher's final predictions. They also force the student's intermediate features to reproduce the teacher's internal activations. This practice is called feature distillation. Almost all such methods count every teacher channel equally in that matching loss, yet the teacher's channels are not equally useful. Work on transferring pretrained features to few-shot tasks shows that many channels carry mostly within-class variation. Down-weighting them, using a score based on how well each channel separates classes relative to how much it varies inside a class, improves accuracy, and keeping only a small fraction of channels often beats keeping all of them (Less is More). A small student has limited capacity. Spending it on reproducing the teacher's within-class noise is waste.

We propose to weight the per-channel feature-distillation loss by a continuous class-separability score, the Fisher ratio, computed once from the frozen teacher on the labelled training set. A single sharpness exponent interpolates between uniform matching and hard channel selection. The testable claim is specific and goes beyond "weighting helps": the best sharpness should grow as the student gets smaller relative to the teacher. The mechanism is that a student with less capacity should concentrate harder on the channels that carry class information. If that holds, it gives a cheap, label-driven handle on the capacity-gap problem, which DPK attacks with a much heavier reconstruction apparatus (Better Teacher Better Student).

Core Claim

Take teacher features at a distilled layer with channels $c=1,\dots,C$, and let $\bar t_{ic}$ be the spatially pooled activation of image $i$ with label $y_i\in\{1,\dots,K\}$. The Fisher ratio of channel $c$ is

$$ r_c=\frac{\sum_k n_k(\mu_{kc}-\mu_c)^2}{\sum_k\sum_{i:y_i=k}(\bar t_{ic}-\mu_{kc})^2}, $$

where $\mu_{kc}$ is the class-$k$ mean, $\mu_c$ the overall mean, and $n_k$ the class count. The distillation loss on the full spatial maps is

$$ \mathcal{L}_{\text{feat}}=\sum_{c=1}^{C} w_c\,\lVert T_c-g(S)_c\rVert_2^2,\qquad w_c=C\,\frac{r_c^{\gamma}}{\sum_{c'} r_{c'}^{\gamma}}. $$

Here $T_c$ is the teacher's channel-$c$ map, $g$ is the channel-wise translator of Liu et al. applied to student features $S$, and $\gamma\ge 0$ is the sharpness. The weights average to one, so the loss scale matches the unweighted baseline. Setting $\gamma=0$ recovers Liu et al. exactly. As $\gamma$ grows, the loss concentrates on a few channels and approaches hard top-$k$ selection of the kind CLDFD uses at test time (CLDFD). The prediction is that the best $\gamma^*$ increases monotonically as the student-to-teacher parameter ratio falls. It also predicts a gain over $\gamma=0$ at every size, with the largest gain for the smallest students.

Relation to Prior Work

Four papers motivate this proposal. Less is More supplies the criterion: its Oracle analysis ties optimal channel weighting to the ratio of between-class separation to within-class variance. Its open problem is that this ratio cannot be estimated reliably from a handful of support images. Distillation removes that problem, because the teacher's full labelled training set is available. We reuse the criterion in a regime where it can be estimated accurately. He et al. find that intermediate alignment drives most distillation gains. Nouriborji et al. find that matching hidden states beats matching attention maps. Both results justify putting the intervention in the feature-matching loss rather than elsewhere.

The closest existing method is DCSF-KD. It also weights a per-channel distillation loss continuously, but its weights come from raw teacher activation magnitude, computed as a global average pool followed by a softmax, and it is evaluated only on detection. Magnitude and discriminability can disagree: a channel can fire strongly on every image and still carry no class information. Our method replaces magnitude with a class-conditional statistic. The magnitude-versus-Fisher comparison is our main ablation.

Several other methods also weight features during distillation, but in different places. DMKD uses continuous teacher attention only to decide what to mask for reconstruction, and it balances its two branches with global scalars. AFD weights which teacher layer each student layer matches, but it matches all channels uniformly within a pair. TOFD filters for task relevance through auxiliary classifier branches. Those branches cost training memory and degrade when four or more are added. None of these weights channels by class separability.

One paper constrains the design. Tian et al. show that in Vision Transformers each token spreads its energy broadly across channels, and the teacher's dominant subspace rotates from image to image. They find that plain MSE feature distillation from CaiT-S24 into DeiT-Tiny gives no gain at all. This suggests that axis-aligned channel weights may be the wrong unit for transformers. We therefore test a variant that computes Fisher weights in the eigenbasis of the teacher's between-class scatter rather than on raw channels. We also treat ViTs as a stress test where failure of the channel-basis version is a live possibility.

Research Program

The weights are computed in one pass of the frozen teacher over the training set, so the added training cost is negligible and the inference cost is zero. We plug the weighted loss into the Liu et al. translator pipeline, alongside standard logit distillation, for convolutional pairs on CIFAR-100 and ImageNet-1K.

The central experiment is a sweep over $\gamma$ at several student sizes with a fixed teacher. On CIFAR-100 we distil a ResNet56 teacher into ResNet20 and narrower ResNets. On ImageNet we distil ResNet-34 and ResNet-152 teachers into ResNet-18 and MobileNetV1. From this sweep we estimate $\gamma^*$ as a function of the capacity ratio.

Two analyses test whether the mechanism is what we claim. First, we measure how much of the student's final-layer within-class variance falls in teacher-low-Fisher directions, with and without weighting. The student should inherit less of the teacher's within-class spread. Second, we check whether gains depend on the layer: we expect deep layers to have more peaked Fisher distributions and to benefit more. Finally, we extend to ViTs following the ViTKD recipe, which mimics shallow FFN outputs and reconstructs masked deep tokens. There we compare channel-basis weights with the rotated between-class-scatter basis.

Baselines

The naive reference points are the student trained alone and the student trained with logit distillation only. For CIFAR-100 with ResNet56→ResNet20, AFD reports 70.98% for standard KD.

The unweighted baseline is Liu et al.'s channel-wise translator with plain $L_2$ loss. This is exactly our method at $\gamma=0$, so it isolates the weighting as the only change. That paper reports ImageNet-1K top-1 of 72.49% for MobileNetV1, up from 69.21%. We run it with identical translator, schedule, data augmentation and logit-loss weight, and we retune only its loss coefficient under the same search budget we give ourselves.

The strongest competitors on this question fall in two groups. The first is methods that address the same capacity-gap and noise-transfer problems. DPK reports 72.51% ImageNet top-1 for ResNet-34→ResNet-18 and 73.29% with a ResNet-152 teacher. It is the strongest competitor specifically on the capacity-gap claim. The second is methods that filter or weight what gets distilled. AFD reports 71.53% on CIFAR-100 ResNet56→ResNet20. TOFD reports a 5.46% average gain across CIFAR-100 architectures, but the file gives no single-pair number, so we will rerun it on our pairs.

For the ViT extension, ViTKD reports 76.06% for DeiT-Tiny alone and 77.78% with logit KD. We will add our weighting on top of that recipe. Tian et al. provide the 74.86% reference for plain MSE from CaiT-S24 into DeiT-Tiny.

Three ablations isolate the key ingredient. The first uses DCSF-KD-style magnitude weights, sharpened to the same weight entropy as ours. The second uses our Fisher weights randomly permuted across channels, which keeps the weight distribution and tests whether any peaking helps. The third uses hard top-$k$ channel selection with $k$ equal to the effective number of channels of our weights. For DCSF-KD on classification, no reported result was found on Lacuna, so we run it ourselves.

Beating the unweighted baseline by a margin larger than the seed-to-seed spread over three seeds, at every student size, counts as success. So does matching or exceeding DPK on ImageNet ResNet-18 without DPK's encoder-decoder. Beating the magnitude and permuted ablations is required before crediting the class-conditional statistic.

Why not just…?

Why not just use activation-magnitude weights as DCSF-KD does? Magnitude-based weights are label-free and already published, so if they work equally well our statistic adds nothing. Our view is that high-magnitude channels in pretrained networks are often the ones carrying broad, class-agnostic responses. That is the channel-bias phenomenon of Less is More. The entropy-matched magnitude ablation settles this directly. If it ties Fisher weighting, we will report that the gain comes from peaking the loss, not from class information.

Why not just hard-select the top-$k$ channels? Hard selection is simpler, and CLDFD shows that a top-$h$ filter can help. However, that filter needs a domain-specific $h$ and discards partially informative channels entirely. Soft masks retain partial information that binary masks destroy, as found for occlusion (End2End Occluded Face Recognition) and for few-shot patch filtering (SAFF). The matched top-$k$ ablation tests this in distillation. If hard selection ties at large $\gamma^*$, we will reframe the method as a principled way to choose $k$.

Why not just learn the channel weights end to end? Weights learned by minimizing the distillation loss collapse toward whichever channels are easiest to match, which is the opposite of what we want. Adding a task loss to prevent collapse reintroduces TOFD-style multi-objective balancing. Fixed teacher-derived weights avoid both problems. We will still run one learned-weights variant with the normalization constraint to show whether collapse actually occurs.

Why not just widen the student, as Tian et al. do? WideLast recovers +3.37% for DeiT-Tiny, but it enlarges the deployed model. Our method adds nothing at inference. For transformers this alternative is a real threat. If even the rotated-basis weights fail there, the honest conclusion is that ViT encoding mismatch needs extra capacity, and that our method applies mainly to convolutional students.

Decisive Checks

Three results would falsify the core claim, each on CIFAR-100 and ImageNet-1K with top-1 accuracy over three seeds. The first is if the Fisher-weighted loss fails to beat both the $\gamma=0$ translator and the entropy-matched magnitude weights. The second is if $\gamma^*$ does not increase as the student shrinks relative to a fixed teacher. The third is if distilled students do not show reduced within-class variance along the teacher's low-Fisher directions.

A secondary check measures whether the teacher-scaling result holds. When moving from a ResNet-34 to a ResNet-152 teacher for ResNet-18, the student should improve under Fisher weighting, as it does under DPK, rather than degrade. Transfer to CUB200 and MS COCO detection, the latter using per-channel pooled statistics over ground-truth boxes, tests whether the benefit survives beyond the classification label set used to compute the weights.