Distill the Distances the Teacher Is Sure About

Relational knowledge distillation trains a small "student" network to copy how a large "teacher" network arranges examples relative to each other: if the teacher places two images close together, the student should too. Copying this geometry, rather than only the teacher's output probabilities, lets students match or even beat their teachers on retrieval tasks. The method gets weaker when the teacher's representation is very high-dimensional, the typical case for modern 768- to 2048-wide layers. The usual explanation is distance concentration: in high dimensions almost every pair of points sits at roughly the same distance, so the teacher seems to have little relational structure left to pass on.

This note argues that the usual framing points to the wrong fix. If concentration only shrank the spread of distances, rescaling them would restore the training signal, and that costs one line of code. We hypothesize that the real damage is different. Inside the concentrated band, the teacher's distances are unreliable: a small change to the input, such as a crop or color jitter, reorders them. A student trained to match every pair equally therefore spends its capacity copying teacher noise. We propose to estimate, for each pair, how stable the teacher's distance is across augmentations, and to distill only in proportion to that stability. The main test is whether this beats a properly rescaled baseline, and whether the gap grows as we raise the teacher's effective dimensionality in a controlled way.

Core Claim

Relational Knowledge Distillation (RKD) matches pairwise teacher and student distances after dividing each set by its batch mean, and matches angles formed by triplets of examples. Its strongest result is a ResNet18 student reaching 82.50% Recall@1 on Cars 196, above its ResNet50 teacher's 77.17%, with a 512-d teacher distilled into a 128-d student. The theory paper Cluster-aware Semi-supervised Learning shows that RKD on unlabeled data amounts to spectral clustering of a graph built from the teacher's distances. Its authors name blurred cluster separation in high-dimensional hidden layers as an open limitation. Together these papers motivate the problem. Neither says what in the teacher's geometry actually goes wrong.

Our claim has two parts. The first is diagnostic. As the teacher's effective dimension grows, two quantities change: the relative spread of pairwise distances shrinks, and the fraction of pairs whose ordering survives augmentation drops. RKD accuracy should track the second quantity more closely than the first.

The second part is a method. Let $\bar D_{ij}$ be the mean teacher distance between examples $i$ and $j$ over $K$ augmented views of each, and let $\sigma_{ij}^2$ be its variance over those views. Let $\mu$ and $s$ be the batch mean and standard deviation of $\bar D$. We weight each pair by

$$w_{ij} = \frac{(\bar D_{ij}-\mu)^2}{(\bar D_{ij}-\mu)^2 + \sigma_{ij}^2},$$

which is the share of the pair's squared deviation from the batch mean that is not explained by augmentation noise. This has the same form as a reliability coefficient. Pairs that stand out from the concentrated band and stay put when the input is perturbed receive weight near 1. Pairs inside the band whose distance is mostly noise receive weight near 0. The student loss is

$$L = \sum_{i,j} w_{ij}\, \ell_\delta\!\left(\frac{\bar D_{ij}-\mu}{s},\; \frac{D^{S}_{ij}-\mu_S}{s_S}\right),$$

where $D^S_{ij}$ is the student distance, $\mu_S$ and $s_S$ are its batch mean and standard deviation, and $\ell_\delta$ is RKD's Huber loss. Both sides are standardized before matching. This is deliberate: the scale problem is handled by the trivial fix inside the loss, so any gain must come from the weighting.

The weighting keeps far-but-stable pairs as well as near neighbors. That matters because the spectral-clustering view of RKD needs between-cluster edges, not only within-cluster ones.

This differs from the closest work in specific ways. RKD weights all pairs equally and corrects only for scale. Sampling Matters in Deep Embedding Learning also counters concentration, by sampling negatives in inverse proportion to the distance density on the sphere. Its correction uses ground-truth labels and an analytic density that depends on dimension, and it says nothing about whether a teacher's distances can be trusted. TRIM uses per-vector local landmarks to rescue triangle-inequality pruning under concentration, but only at search time. We borrow its intuition that local reference points stay informative after global distances have concentrated, and apply it during training.

Research Program

The first component is a measurement study. For each teacher we compute three statistics on held-out data at several layers. The first is relative contrast $(D_{\max}-D_{\min})/D_{\min}$, the classical concentration quantity from multi-dimensional similarity search. The second is the coefficient of variation of pairwise distances. The third is the fraction of pairs whose distance rank stays within a fixed tolerance across augmentations. Learned representations usually have intrinsic dimension far below their width, so we expect some wide layers to show little concentration. We will report that honestly, because it limits where the method can help.

The second component is a controlled knob for dimensionality. We append $m$ independent Gaussian coordinates to frozen teacher features. Their variance is matched to the teacher's per-coordinate variance and they are resampled for each augmented view. This raises ambient dimension and concentration while leaving the semantic structure unchanged. Because the injected coordinates change under augmentation, they are exactly the kind of noise the reliability weight is meant to detect.

We will also check a second regime that the noise knob does not cover. Fixed random padding (the same random vector for each example across all views) also concentrates distances but is stable under augmentation. We expect the weighting to help less there. That result would show which mechanism the method addresses.

The third component is the method itself and its cost. Re-running the teacher on $K$ views per step multiplies teacher cost by $K$. To avoid this, we store each example's $K$ teacher embeddings in an offline bank, in the spirit of Contrastive Representation Distillation's memory buffer. We use $K=4$ by default and ablate $K\in\{2,8\}$. We will also test the weighting on RKD's triplet angle loss, using angle stability as the reliability measure.

Baselines

The naive reference points are a student trained on labels alone and a student trained with standard temperature-scaled logit distillation. These anchor how much any relational term adds. On CIFAR-100 the RKD paper reports logit distillation at 74.26% top-1 and logit distillation plus RKD at 74.66%.

The primary baseline is RKD as published, combining distance and angle losses with batch-mean normalization. On Cars 196 we use its reported 82.50% Recall@1 for a 512-d ResNet50 teacher and 128-d ResNet18 student as a sanity target for our reimplementation. On CIFAR-10 with 40 labels, Cluster-aware Semi-supervised Learning reports FixMatch plus RKD at 91.19% against FixMatch alone at 85.48%. That setting is where the authors flagged the high-dimensional limitation, so it is our main semi-supervised test bed.

The strongest competing representation-distillation method is CRD, which reports 76.05% top-1 on CIFAR-100 for a WideResNet-40-2 teacher and ShuffleNetV1 student, against 74.83% for logit distillation. We also run CRD with its memory-bank negatives drawn by distance-weighted sampling, adapted from Sampling Matters in Deep Embedding Learning. This is the obvious concentration-aware version of the strongest competitor. For distance-weighted sampling on its own, the file reports 61.7% Recall@1 on Stanford Online Products. On CUB-200-2011 we found no reported result on Lacuna for RKD, so all CUB numbers will come from our own runs.

The ablations that isolate the key ingredient are three RKD variants built from the same code. The first is z-scored RKD, which is our loss with all $w_{ij}=1$. The second is rank-matched RKD, which converts teacher distances to within-batch quantiles; this is equivalent to remapping the teacher's distance distribution onto any target shape, including one matched to the student's dimension. The third is top-$k$ neighbor RKD, which keeps only each anchor's $k$ nearest teacher neighbors.

All methods share teacher checkpoints, student architectures, augmentation pipelines, epochs, and batch sizes. Each method gets the same tuning budget of 20 trials over its own loss weights and temperatures, chosen on a held-out validation split. Results are averaged over three seeds. Beating the competitors means two things. Our method must exceed z-scored RKD and CRD with distance-weighted sampling by more than the seed spread on the semi-supervised CIFAR-10 setting and on Cars 196. Its advantage over z-scored RKD must also grow monotonically with the number of injected noise dimensions.

Why not just…?

Why not just standardize the distances instead of weighting them? This is the most serious threat, and it is why z-scored RKD is the ablation that matters most. If concentration only compresses scale, standardization fully repairs the signal and our weight adds nothing. The noise-injection sweep settles the question. Augmentation-varying noise dimensions leave standardized distances well scaled but increasingly unreliable. Our hypothesis predicts that z-scored RKD degrades along this sweep while the weighted loss does not. If the two curves coincide, the hypothesis is wrong, and the lesson to publish is that normalizing by the batch mean was RKD's only real problem.

Why not just project features onto the unit sphere and match cosines? Sampling Matters in Deep Embedding Learning shows that distances between random points on a high-dimensional sphere still concentrate tightly around $\sqrt{2}$. SEN reports that hard normalization causes optimization trouble and is outperformed by soft norm penalties in few-shot learning: 54.6% against 52.8% for Ring Loss on FC100. We include cosine RKD as a variant, but neither result suggests it removes unreliable pairs.

Why not just reduce the teacher to fewer dimensions with PCA first? PCA removes low-variance directions, not unstable ones. Augmentation noise can be high-variance. Separately, Maximum Variance Unfolding reports that PCA needs far more dimensions than a nonlinear method to keep 95% of variance on curved data: 59 against 2 on Teapots and 80 against 4 on Faces. We run PCA-to-student-width followed by z-scored RKD. If it matches our method on real teachers, the cheaper recipe should be preferred, and we will say so.

Why not just distill nearest neighbors only? Keeping each example's top-$k$ teacher neighbors removes most of the concentrated band at no extra cost. It also discards stable far pairs that encode between-cluster separation, which the spectral-clustering analysis of RKD relies on. The top-$k$ ablation, with $k$ tuned under the same budget, tests whether those pairs matter. We will look most closely at the 40-label setting, where cluster structure matters most.

Proposed Experiments

The first experiment is the diagnostic correlation. We distill students from teacher layers of increasing width: 512 and 2048 from ResNet50, and 384 and 768 from vision transformers of the kind used in Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation. We then ask whether RKD's gain over logit distillation correlates better with relative contrast or with augmentation rank stability.

The second experiment is the noise-dimension sweep. We add $m \in \{0, 256, 1024, 4096\}$ coordinates to a 512-d teacher, run it on Cars 196 and on CIFAR-10 with 40 labels, and evaluate all relational variants. We repeat the sweep with stable random padding as the contrasting regime described above.

The third experiment covers the full benchmarks: Cars 196, CUB-200-2011, and Stanford Online Products for retrieval, measured by Recall@1; CIFAR-100 and CIFAR-10 with 40 labels for classification, measured by top-1 accuracy; and an ImageNet ResNet-34 to ResNet-18 run if the cheaper settings justify the cost. We report training time with and without the embedding bank. RKD noted that its students lose out-of-distribution transfer, so we also evaluate Cars-trained students on CUB, to see whether downweighting noisy pairs reduces this over-specialization.

The decisive check is the second experiment. A widening gap between reliability-weighted and z-scored RKD as noise dimensions are added would support the claim that concentration harms distillation by degrading reliability rather than scale. A flat gap would refute it.